Search

Great progress has been witnessed for weakly supervised semantic segmentation, which aims to segment objects without dense pixel annotations. Most approaches concentrate on generating high quality pseudo labels, which are then fed into a standard segmentation model as supervision. However, such a solution has one major limitation: noise of pseudo labels is inevitable, which is unsolvable for the standard segmentation model. In this paper, we propose a credible dual-expert learning (CDL) framework to mitigate the noise of pseudo labels. Specifically, we first observe that the model predictions with different optimization loss functions will have different credible regions; thus, it is possible to make self-corrections with multiple predictions. Based on this observation, we design a dual-expert structure to mine credible predictions, which are then processed by our noise correction module to update pseudo labels in an online way. Meanwhile, to handle the case that the dual-expert produces incredible predictions for the same region, we design a relationship transfer module to provide feature relationships, enabling our noise correction module to transfer predictions from the credible regions to such incredible regions. Considering the above designs, we propose a base CDL network and an extended CDL network to satisfy different requirements. Extensive experiments show that directly replacing our model with a conventional fully supervised segmentation model, the performances of various weakly supervised semantic segmentation pipelines were boosted, achieving new state-of-the-art performances on both PASCAL VOC 2012 and MS COCO with a clear margin. Code will be available at: https://github.com/zbf1991/CDL. [ABSTRACT FROM AUTHOR]

Weakly supervised semantic segmentation is receiving great attention due to its low human annotation cost. In this paper, we aim to tackle bounding box supervised semantic segmentation, i.e., training accurate semantic segmentation models using bounding box annotations as supervision. To this end, we propose affinity attention graph neural network ($A^2$ A 2 GNN). Following previous practices, we first generate pseudo semantic-aware seeds, which are then formed into semantic graphs based on our newly proposed affinity Convolutional Neural Network (CNN). Then the built graphs are input to our $A^2$ A 2 GNN, in which an affinity attention layer is designed to acquire the short- and long- distance information from soft graph edges to accurately propagate semantic labels from the confident seeds to the unlabeled pixels. However, to guarantee the precision of the seeds, we only adopt a limited number of confident pixel seed labels for $A^2$ A 2 GNN, which may lead to insufficient supervision for training. To alleviate this issue, we further introduce a new loss function and a consistency-checking mechanism to leverage the bounding box constraint, so that more reliable guidance can be included for the model optimization. Experiments show that our approach achieves new state-of-the-art performances on Pascal VOC 2012 datasets (val: 76.5 percent, test: 75.2 percent). More importantly, our approach can be readily applied to bounding box supervised instance segmentation task or other weakly supervised semantic segmentation tasks, with state-of-the-art or comparable performance among almot all weakly supervised tasks on PASCAL VOC or COCO dataset. Our source code will be available at https://github.com/zbf1991/A2GNN. [ABSTRACT FROM AUTHOR]

Language-person search is an essential technique for applications like criminal searching, where it is more feasible for a witness to provide language descriptions of a suspect than providing a photo. Most existing works treat the language-person pair as a black-box, neither considering the inner structure in a person picture, nor the correlations between image regions and referring words. In this work, we propose a transformer-based language-person search framework with matching conducted between words and image regions, where a person picture is vertically separated into multiple regions using two different ways, including the overlapped slicing and the key-point-based slicing. The co-attention between linguistic referring words and visual features are evaluated via transformer blocks. Besides the obtained outstanding searching performance, the proposed method enables to provide interpretability by visualizing the co-attention between image parts in the person picture and the corresponding referring words. Without bells and whistles, we achieve the state-of-the-art performance on the CUHK-PEDES dataset with Rank-1 score of 57.67% and the PA100K dataset with mAP of 22.88%, with simple yet elegant design. Code is available on https://github.com/detectiveli/T-MRS. [ABSTRACT FROM AUTHOR]

In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21 percent), RefCOCO+ (39.18 percent) and RefCOCOg (43.24 percent) datasets, that is 4.17, 4.08 and 7.8 percent higher than the previous one, respectively. The code is available at https://github.com/insomnia94/DTWREG. [ABSTRACT FROM AUTHOR]

The Coarse-To-Fine (CTF) matching scheme has been widely applied to reduce computational complexity and matching ambiguity in stereo matching and optical flow tasks by converting image pairs into multi-scale representations and performing matching from coarse to fine levels. Despite its efficiency, it suffers from several weaknesses, such as tending to blur the edges and miss small structures like thin bars and holes. We find that the pixels of small structures and edges are often assigned with wrong disparity/flow in the upsampling process of the CTF framework, introducing errors to the fine levels and leading to such weaknesses. We observe that these wrong disparity/flow values can be avoided if we select the best-matched value among their neighborhood, which inspires us to propose a novel differentiable Neighbor-Search Upsampling (NSU) module. The NSU module first estimates the matching scores and then selects the best-matched disparity/flow for each pixel from its neighbors. It effectively preserves finer structure details by exploiting the information from the finer level while upsampling the disparity/flow. The proposed module can be a drop-in replacement of the naive upsampling in the CTF matching framework and allows the neural networks to be trained end-to-end. By integrating the proposed NSU module into a baseline CTF matching network, we design our Detail Preserving Coarse-To-Fine (DPCTF) matching network. Comprehensive experiments demonstrate that our DPCTF can boost performances for both stereo matching and optical flow tasks. Notably, our DPCTF achieves new state-of-the-art performances for both tasks - it outperforms the competitive baseline (Bi3D) by 28.8% (from 0.73 to 0.52) on EPE of the FlyingThings3D stereo dataset, and ranks first in KITTI flow 2012 benchmark. The code is available at https://github.com/Deng-Y/DPCTF. [ABSTRACT FROM AUTHOR]

Correlation filter has been proven to be an effective tool for a number of approaches in visual tracking, particularly for seeking a good balance between tracking accuracy and speed. However, correlation filter-based models are susceptible to wrong updates stemming from inaccurate tracking results. To date, very little effort has been devoted towards handling the correlation filter update problem. In this paper, we propose a novel approach to address the correlation filter update problem. In our approach, we update and maintain multiple correlation filter models in parallel, and we use deep reinforcement learning for the selection of an optimal correlation filter model among them. To facilitate the decision process in an efficient manner, we propose a decision-net to deal with target appearance modeling, which is trained through hundreds of challenging videos using proximal policy optimization and a lightweight learning network. An exhaustive evaluation of the proposed approach on the OTB100 and OTB2013 benchmarks shows that the approach is effective enough to achieve the average success rate of 62.3% and the average precision score of 81.2%, both exceeding the performance of traditional correlation filter-based trackers. [ABSTRACT FROM AUTHOR]

Video content delivery over Internet is receiving increasing attention from both industry and academia, especially for the multiview video contents, as it is the basis to support various applications, such as 3-D video, virtual reality, free view video, and so on. To cope with the dynamic nature of Internet throughput, dynamic adaptive streaming over HTTP (DASH) has been introduced to control the video streaming based on the network conditions. In this paper, we design a streaming framework to improve the user experience of the multiview video streaming over DASH, considering the user behavior of the viewpoint navigation during the streaming process. To eliminate the view switching delay, a multiple view navigation rule is introduced to pre-fetch the possible switching viewpoints. An optimal bitrate allocation scheme is proposed for the introduced rule, allowing the clients to maximize the video quality. Moreover, we found the video quality and the playback starvation probability are conflicting factors, while both are essential for the user’s quality of experience (QoE). To tackle this issue, a QoE optimization solution is designed to maximize the overall performance in the proposed framework. Several experiments verify the effectiveness of the proposed framework, and the results demonstrate that the proposed framework outperforms two typical DASH methods. [ABSTRACT FROM AUTHOR]

Abstract Person search in real-world scenarios is a new challenging computer version task with many meaningful applications. The challenge of this task mainly comes from: (1) unavailable bounding boxes for pedestrians and the model needs to search for the person over the whole gallery images; (2) huge variance of visual appearance of a particular person owing to varying poses, lighting conditions, and occlusions. To address these two critical issues in modern person search applications, we propose a novel Individual Aggregation Network (IAN) that can accurately localize persons by learning to minimize intra-person feature variations. IAN is built upon the state-of-the-art object detection framework, i.e., faster R-CNN, so that high-quality region proposals for pedestrians can be produced in an online manner. In addition, to relieve the negative effect caused by varying visual appearances of the same individual, IAN introduces a novel center loss that can increase the intra-class compactness of feature representations. The engaged center loss encourages persons with the same identity to have similar feature characteristics. Extensive experimental results on two benchmarks, i.e., CUHK-SYSU and PRW, well demonstrate the superiority of the proposed model. In particular, IAN achieves 77.23% mAP and 80.45% top-1 accuracy on CUHK-SYSU, which outperform the state-of-the-art by 1.7% and 1.85%, respectively. [ABSTRACT FROM AUTHOR]

• We propose a weight-guided class complementing framework to mitigate the gradient shift issue caused by unsampled classes in long-tailed scenario, including a slot-based class complementing strategy and a weight-guided feature mining loss. • We further introduce a weight refining scheme at the end of training process, which can remove the bias term in our weight-guided class complementing framework caused by long-tailed distribution. • Our plug-and-play weight-guided class complementing framework can be easily implemented to different existing approaches, achieving significant improvement on various benchmarks with new state-of-the-art performances. Real-world data are often long-tailed distributed and have plenty classes. This characteristic leads to a significant performance drop for various models. One reason behind that is the gradient shift caused by unsampled classes in each training iteration. In this paper, we propose a W eight- G uided C lass C omplementing framework to address this issue. Specifically, this framework first complements the unsampled classes in each training iteration by using a dynamic updated data slot. Then, considering the over-fitting issue caused by class complementing, we utilize the classifier weights as learned knowledge and encourage the model to discover more class specific characteristics. Finally, we design a weight refining scheme to deal with the long-tailed bias existing in classifier weights. Experimental results show that our framework can be implemented upon different existing approaches effectively, achieving consistent improvements on various benchmarks with new state-of-the-art performances. Codes will be released. [ABSTRACT FROM AUTHOR]

Visual object tracking is a challenging task considering illumination variation, occlusion, rotation, deformation and other problems. In this paper, we extend a Siamese INstance search Tracker (SINT) with model updating mechanism to improve its tracking robustness. SINT uses convolutional neural network (CNN) features, and compares the new frame features with the target features in the first frame. The candidate region with the highest similarity score is considered as the tracking result. However, SINT is not robust against large target variation because the matching model is not updated during the whole tracking process. To combat this defect, we propose an Ensemble Siamese Tracker (EST), where the final similarity score is also affected by the similarity with tracking results in recent frames instead of solely considering the first frame. Tracking results in recent frames are used to adjust the model for continuous target change. Meanwhile, we combine large displacement optical flow method with EST to further improve the performance (called EST+). We test the proposed EST and EST+ on a standard tracking benchmark OTB. It turns out the average overlap ratio of EST and EST+ increase 2.72% and 3.55% respectively compared with SINT on OTB 2013, which contains 51 video sequences. For the OTB 100, the average overlap ratio gain is 4.2%. [ABSTRACT FROM AUTHOR]

The long-tailed characteristic leads to a significant performance drop for various models on long-tailed distribution datasets. Existing works mainly focus on mitigating the data shortage in tail classes at dataset level by data re-sampling, loss re-weighting or knowledge transfer from head to tail. While in this paper, we focus on another perspective which is also related to the performance drop: the gap between total dataset class number and training batch size. To address this issue, we propose a Weight-Guided (WG) loss which utilizes the classifier weights as auxiliary tail samples. It can be easily deployed to different models. By simply adding WG loss to Mask R-CNN with ResNet-50 backbone, we improve the performance by (i) 0.5 box AP and 0.4 mask AP on COCO dataset, (ii) 0.4 box and mask AP (1.8 mask AP for rare classes) on LVIS v1.0 dataset. Codes will be released. • Point out the small class ratio covered in one batch can cause gradient shifting. • Propose a weight-guided loss to overcome the limitation of small batch size. • Our WG loss improves various frameworks consistently, especially for rare classes. [ABSTRACT FROM AUTHOR]

Data distribution alignment and clustering-based self-training are two feasible solutions to tackle unsupervised domain adaptation (UDA) on person re-identification (re-ID). Most existing alignment-based methods solely learn the source domain decision boundaries and align the data distribution of the target domain to the source domain, thus the re-ID performance on the target domain completely depends on the shared decision boundaries and how well the alignment is performed. However, two domains can hardly be precisely aligned because of the label space discrepancy of two domains, resulting in poor target domain re-ID performance. Although clustering-based self-training approaches could learn independent decision boundaries on the pseudo-labelled target domain data, they ignore both the accurate ID-related information of the labelled source domain data and the underlying relations between two domains. To fully exploit the source domain data to learn discriminative target domain ID-related features, in this paper, we propose a novel cross-domain alignment method in the homogeneous distance space, which is constructed by the newly designed stair-stepping alignment (SSA) matcher. Such alignment method can be integrated into both alignment-based framework and clustering-based framework. Extensive experiments validate the effectiveness of our proposed alignment method in these two frameworks. We achieve superior performance when the proposed alignment module is integrated into the clustering-based framework. Codes will be available at: http://github.com/Dingyuan-Zheng/HDS. [ABSTRACT FROM AUTHOR]

• We make an exten sion of our previous wok and design a more powerful end to end n etwork for weakly supervised semantic segmentation • We propose two new loss functions for utilizing the reliable labels, including a new dense energy loss and a batch based class distance loss. The former relies on shallow features, whilst the latter focuses on distinguishing high level s emantic features for different classes. • We design a new attention module to extract comprehensive global information. By using a re weighting technique, it can suppress dominant or noisy attention values and aggregate sufficient global information. • Our approach achieves a new state of the art performance for weakly supervised semantic segmentation. Weakly supervised semantic segmentation is a challenging task that only takes image-level labels as supervision but produces pixel-level predictions for testing. To address such a challenging task, most current approaches generate pseudo pixel masks first that are then fed into a separate semantic segmentation network. However, these two-step approaches suffer from high complexity and being hard to train as a whole. In this work, we harness the image-level labels to produce reliable pixel-level annotations and design a fully end-to-end network to learn to predict segmentation maps. Concretely, we firstly leverage an image classification branch to generate class activation maps for the annotated categories, which are further pruned into tiny reliable object/background regions. Such reliable regions are then directly served as ground-truth labels for the segmentation branch, where both global information and local information sub-branches are used to generate accurate pixel-level predictions. Furthermore, a new joint loss is proposed that considers both shallow and high-level features. Despite its apparent simplicity, our end-to-end solution achieves competitive mIoU scores (val : 65.4%, test : 65.3%) on Pascal VOC compared with the two-step counterparts. By extending our one-step method to two-step, we get a new state-of-the-art performance on the Pascal VOC 2012 dataset(val : 69.3%, test : 69.2%). Code is available at: https://github.com/zbf1991/RRM. [ABSTRACT FROM AUTHOR]

• We changed the title to "Soft Pseudo-Label Shrinkage for Unsupervised Domain Adaptive Person Re-identification". One effective way to tackle unsupervised domain adaptation (UDA) on person re-identification (Re-ID) is to use clustering-based self-training approach, where a model is trained with hard pseudo-labels obtained from a clustering method. Using a hard pseudo-label, a sample is assigned to the cluster with the highest probability, which is sensitive to the incorrect clustering result due to imperfect clustering algorithms. Soft pseudo-labels can mitigate this issue by representing the sample with the full range of class probabilities from all clusters. Specifically, soft pseudo-labels comprise probabilities of full range classes, because they consider both the hard samples and easy samples. This will distract the model from learning more discriminative features in the hard examples. To solve this issue, we propose a coarse-to-fine refinement mechanism to produce robust refined soft pseudo-labels by progressively focusing more on the hard samples while less on the easy samples. The proposed refined soft pseudo-labels can be readily integrated into cross-entropy loss as a strong supervision to guide the model to learn more discriminative features. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art unsupervised domain adaptation approaches on person Re-ID with a considerable margin. Code will be available at: http://github.com/Dingyuan-Zheng/ctf-UDA. [ABSTRACT FROM AUTHOR]

In the multiview video plus depth (MVD) coding format, both texture and depth views are jointly compressed to represent the 3-D video content. The MVD format enables synthesis of virtual views through depth-image-based rendering; hence, distortion in the texture and depth views affects the quality of the synthesized virtual views. Bit allocation between texture and depth views has been studied with some promising results. However, to the best of our knowledge, most of the existing bit-allocation methods attempt to allocate a fixed amount of total bit rate between texture and depth views; that is, to select appropriate pair of quantization parameters for texture and depth views to maximize the synthesized view quality subject to a fixed total bit rate. In this paper we propose a scalable bit-allocation scheme, where a single ordering of texture and depth packets is derived and used to obtain optimal bit allocation between texture and depth views for any total target rates. In the proposed scheme, both texture and depth views are encoded using the quality scalable coding method; that is, medium grain scalable (MGS) coding of the Scalable Video Coding (SVC) extension of the Advanced Video Coding (H.264/AVC) standard. For varying target total bit rates, optimal bit truncation points for both texture and depth views can be obtained using the proposed scheme. Moreover, we propose to order the enhancement layer packets of the H.264/SVC MGS encoded depth view according to their contribution to the reduction of the synthesized view distortion. On one hand, this improves the depth view packet ordering when considered the rate-distortion performance of synthesized views, which is demonstrated by the experimental results. On the other hand, the information obtained in this step is used to facilitate optimal bit allocation between texture and depth views. Experimental results demonstrate the effectiveness of the proposed scalable bit-allocation scheme for texture and depth views. [ABSTRACT FROM AUTHOR]

Forward error correction (FEC) codes are widely studied to protect streamed video over unreliable networks. Typically, enlarging the FEC coding block size can improve the error correction performance. For video streaming applications, this could be implemented by grouping more than one video frame into one FEC coding block. However, in this case, it leads to decoding delay, which is not tolerable for real-time video streaming applications. In this paper, to solve this dilemma, a real-time video streaming scheme using randomized expanding Reed–Solomon (RS) code is proposed. In this scheme, the RS coding block includes not only the video packets of the current frame, but could also include all the video packets of previous frames in the current group of pictures. At the decoding side, the parity-check equations of the current frame are jointly solved with all the parity-check equations of the previous frames. Since video packets of the following frames are not encompassed in the RS coding block, no delay will be caused for waiting for the video or parity packets of the following frames both at encoding and decoding sides. Experimental results show that the proposed scheme outperforms other real-time error resilient video streaming approaches significantly, specifically, for the Foreman sequence, the proposed scheme could provide 1.5 dB average gain over the state-of-the-art approach for 10% i.i.d. packet loss rate, whereas for the burst loss case, the average gain is more than 3 dB.ref refid="fn1"/ id="fn1" asterisk="no"paraMATLAB code of this paper is available for download at http://www.mmtlab.com.para [ABSTRACT FROM PUBLISHER]

For real-time video streaming systems, the video packets arriving after the display deadline of their frames are considered as late-arrival packets, and typically they are discarded. This will affect the current frame and the following ones due to error propagations. For this reason, in this paper, we propose an approach to exploit the late-arrival and out-of-order packets, which includes two mechanisms. The first mechanism will use these packets to update the reference frames to make them more consistent with the encoder side, and this will eventually reduce the error propagations. The second mechanism will use these packets to increase the chance of successfully decoding the Reed-Solomon (RS) code. In the proposed approach, a sub-GOP based systematic RS code is used and optimized to exploit these packets, where the size of each sub-GOP and the parity packet number for each sub-GOP are optimally tuned, taking into consideration the maximum end-to-end delay, the network conditions, and other system parameters, so as to make the best use of the late-arrival packets and to exploit the out-of-order packets. Finally, the experimental results show the advantage of the proposed approach over other approaches. [ABSTRACT FROM AUTHOR]

Video object segmentation, aiming to segment the foreground objects given the annotation of the first frame, has been attracting increasing attentions. Many state-of-the-art approaches have achieved great performance by relying on online model updating or mask-propagation techniques. However, most online models require high computational cost due to model fine-tuning during inference. Most mask-propagation based models are faster but with relatively low performance due to failure to adapt to object appearance variation. In this paper, we are aiming to design a new model to make a good balance between speed and performance. We propose a model, called NPMCA-net, which directly localizes foreground objects based on mask-propagation and non-local technique by matching pixels in reference and target frames. Since we bring in information of both first and previous frames, our network is robust to large object appearance variation, and can better adapt to occlusions. Extensive experiments show that our approach can achieve a new state-of-the-art performance with a fast speed at the same time (86.5% IoU on DAVIS-2016 and 72.2% IoU on DAVIS-2017, with speed of 0.11s per frame) under the same level comparison. Source code is available at https://github.com/siyueyu/NPMCA-net. • NPMCA-net strikes a good balance between accuracy and running speed. • Our method aggregates multi-frame information with high level interpretability. • Our methods achieves new state-of-the-art performances on reported datasets. [ABSTRACT FROM AUTHOR]

• We identify the necessity of triplet loss in image based one-shot Re-ID, where the use of noisy pseudo labels for training is inevitable. Considering the nature of pseudo labels, we introduce an HSoften-Triplet-Loss to soften the negative influence of incorrect pseudo label. Meanwhile, a new batch formation rule is designed by taking different nature of labelled samples and pseudo labelled samples into account. • We propose a pseudo label sampling mechanism for one-shot Re-ID task, which is based on the relative sample distance to the feature center of each labelled sample. Our sampling mechanism ensures the feasibility of forming a positive pair and a negative pair samples for each class label, which paves the way for the utilization of the HSoften-Triplet-Loss. • We achieve the state-of-the-art mAP score of 42.7% on Market1501 and 40.3% on DukeMTMC-Reid, 16.5 and 11.8 higher than EUG respectively. In this paper, we aim to tackle the one-shot person re-identification problem where only one image is labelled for each person, while other images are unlabelled. This task is challenging due to the lack of sufficient labelled training data. To tackle this problem, we propose to iteratively guess pseudo labels for the unlabelled image samples, which are later used to update the re-identification model together with the labelled samples. A new sampling mechanism is designed to select unlabelled samples to pseudo labelled samples based on the distance matrix, and to form a training triplet batch including both labelled samples and pseudo labelled samples. We also design an HSoften-Triplet-Loss to soften the negative impact of the incorrect pseudo label, considering the unreliable nature of pseudo labelled samples. Finally, we deploy an adversarial learning method to expand the image samples to different camera views. Our experiments show that our framework achieves a new state-of-the-art one-shot Re-ID performance on Market-1501 (mAP 42.7%) and DukeMTMC-Reid dataset (mAP 40.3%). Code is available on https://github.com/detectiveli/PSMA. [ABSTRACT FROM AUTHOR]

• We deploy the "actor critic" RL framework to train the agent for generating the adaptation areas. • W e further design a novel multi branch tree based exploration method to fast select the best state action pairs to speed up the RL training. • New state of the art result of mean re gion similarity is obtained for the DAVIS 2016 dataset, which is 87.1%. The task of the proposed method is semi-supervised video object segmentation where only the ground-truth segmentation of the first frame is provided. The existing approaches rely on selecting the region of interest for model update; however it is rough and inflexible, leading to performance degradation. To overcome this limitation, a novel approach is proposed which utilizes reinforcement learning to select optimal adaptation areas for each frame, based on the historical segmentation information. The RL model learns to take optimal actions to adjust the region of interest inferred from the previous frame for online model updating. To speed up the model adaption, a novel multi-branch tree based exploration method is designed to quickly select the best state action pairs. The proposed method is evaluated on three common video object segmentation datasets including DAVIS 2016, SegTrack V2 and Youtube-Object. The results show that the proposed work improves the state-of-the-art of the mean region similarity to 87.1% on the DAVIS 2016 dataset, and to 79.5% on the Youtube-Object dataset. Meanwhile, competitive performance is obtained on the SegTrack V2 dataset. Code is at https://github.com/insomnia94/ARG. [ABSTRACT FROM AUTHOR]

Person search aims to search for a target person among multiple images recorded by multiple surveillance cameras, which faces various challenges from both pedestrian detection and person re-identification. Besides the large intra-class variations owing to various illumination conditions, occlusions and varying poses, background clutters in the detected pedestrian bounding boxes further deteriorate the extracted features for each person, making them less discriminative. To tackle these problems, we develop a novel approach which guides the network with segmentation masks so that discriminative features can be learned invariant to the background clutters. We demonstrate that joint optimization of pedestrian detection, person re-identification and pedestrian segmentation enables to produce more discriminative features for pedestrian, and consequently leads to better person search performance. Extensive experiments on two widely used benchmark datasets prove the superiority of our approach. In particular, our proposed model achieves the state-of-the-art performance (86.3% mAP and 86.5% top-1 accuracy) on CUHK-SYSU dataset. [ABSTRACT FROM AUTHOR]

The past decade has witnessed fast development in multiview 3D video technologies, such as Three-Dimensional Video (3DV), Virtual Reality (VR), and Free Viewpoint Video (FVV). However, large information redundancy and a vast amount of multiview video data needs to be stored or transmitted, which poses a serious problem for multiview video systems. Asymmetric multiview video compression can alleviate this problem by coding views with different qualities. Only several viewpoints are kept with high-quality and other views are highly compressed to low-quality. However, highly compressed views may incur severe quality degradation. Thus, it is necessary to enhance the visual quality of highly compressed views at the decoder side. Exploiting similarities among the multiview images is the key to efficiently reconstruct the multiview compressed views. In this paper, we propose a novel method for multiview quality enhancement, which directly learns an end-to-end mapping between the low-quality and high-quality views and recovers the details of the low-quality view. The mapping process is realized using a deep convolutional neural network (MVENet). MVENet takes a low-quality image of one view and a high-quality image of another view of the same scene as inputs and outputs an enhanced image for the low-quality view. To the best of our knowledge, this is the first work for multiview video enhancement where neither a depth map nor a projected virtual view is required in the enhancement process. Experimental results on both computer graphic and real datasets demonstrate the effectiveness of the proposed approach with a peak signal-to-noise ratio (PSNR) gain of up to 2dB over low-quality compressed views using HEVC and up to 3.7dB over low-quality compressed views using JPEG on the benchmark Cityscapes. • Multiview video quality enhancement without depth information approach. • The MVENet network improves the views compressed using HEVC and JPEG. • Only information from neighboring views is required for the CNN. • A novel framework that can directly enhance low-quality views. [ABSTRACT FROM AUTHOR]

DASH, the Dynamic adaptive video streaming over hypertext transfer protocol (HTTP), has become the de-facto video delivery mechanism nowadays, which takes advantage of the existing low cost and wide-spread HTTP platforms. Standards like MPEG-DASH defines the bitstreams conformance and decoding process, while leaving the bitrate adaptive algorithm open for research. So far, most DASH researches focus on the constant bitrate video delivery. In this paper, various bitrate (VBR) video delivery is investigated in the on-demand streaming scenario. Detailed instant bitrates of future segments are exploited in the proposed adaptation method to grasp the fluctuation traits of the VBR video. Meanwhile, the adaptation problem is formulated as an optimization process with the proposed internal QoE goal function, which keeps a good balance between various requirements. Besides, the parameters within the internal QoE function can be tuned to guarantee the flexibility of meeting different preferences. The experimental results demonstrate that our proposed QoE-based video adaptation method outperforms the state-of-the-art method with a good margin. [ABSTRACT FROM PUBLISHER]

Directly processing 3D point cloud data becomes dominant in classification and segmentation tasks. Present mainstream point based methods usually focus on learning in either geometric space (i. e. PointNet++) or semantic space (i. e. DGCNN). Owing to the irregular and unordered data property of point cloud, these methods still suffer from drawbacks of either ambiguous local features aggregation in geometric space or poor global features extraction in semantic space. While few prior works address these two defects simultaneously by fusing information from the dual spaces, we make a first attempt to develop a synergistic framework, called PointGS. Leveraging both the strength of geometric structure and semantic representation, PointGS establishes a mutual supervision mechanism that can bridge the two spaces and fuse complementary information for better analyzing 3D point cloud data. Compared with existing popular networks, our work attains obvious performance improvement on all three mainstream tasks without any sophisticated operations. • We make a first attempt to develop a fusion framework in point cloud analysis. • We build multi-scale geometric inputs and eschew pooling in semantic space. • We explore a new updating rule for simplifying semantic feature aggregation. [ABSTRACT FROM AUTHOR]

Multiview video plus depth is a popular 3D video format which can provide viewers a vivid 3D feeling. However, its requirements in terms of computational complexity and transmission bandwidth are more than that of conventional 2D video. To mitigate these limitations, some works have proposed to reduce the amount of transmitted data by adopting different resolutions for different views, and consequently, the transmitted video is called mixed resolution video. In order to further reduce the transmitted data and maintain good quality at the decoder side; in this paper, we propose a down/upsampling algorithm for 3D multiview video which systematically takes into account the video encoder and decoder. At the encoder side, the rows of the two adjacent views are downsampled following an interlacing and complementary fashion, whereas, at the decoder side, the discarded pixels are recovered by fusing the virtual view pixels with the directional interpolated pixels from the complementary downsampled views. Moreover, the patterns of the texture surrounding the discarded pixels are used to aid the data fusion, so as to enhance edges recovery. Meanwhile, with the assistance of virtual views, at the decoder side, the proposed approach can effectively recover the discarded high-frequency details. The experimental results demonstrate the superior performance of the proposed framework. [ABSTRACT FROM AUTHOR]

Scribble annotations have recently become popular in video salient object detection. Previous methods only focus on utilizing shallow feature consistency for more integral predictions. However, there is potential for consistency between cross-frame deep features to be used to help regularize saliency predictions better. Besides, we have observed that leveraging saliency predictions as pseudo-supervision signals yields notable improvements in extracting both intra-frame and cross-frame deep features. This, in turn, leads to more precise and detailed object structural information. Thus, we propose a cross-frame feature-saliency mutual reinforcing training process to assist scribble annotations for integral video saliency predictions. Specifically, we design a cross-frame feature regularization head, which leverages intra-frame and cross-frame deep feature consistency to regularize saliency predictions as auxiliary supervision. Then, to help obtain more accurate feature consistency, we design a cross-frame saliency regularization head, where predicted saliency values are used as pseudo-supervision signals to acquire better feature consistency. In this way, our cross-frame feature and saliency regularization heads can benefit from each other to help the network learn more accurately. Extensive experiments show that our method can achieve better performances than the previous best methods. The project is available at https://github.com/muchengxue0911/CFMR. • A mutual reinforcing process is proposed to learn comprehensive object structure. • Intra-/cross-frame feature consistency is deployed to supervise saliency predictions. • Predicted saliency values are used to help build a more accurate feature consistency. [ABSTRACT FROM AUTHOR]

This work aims at active domain adaptation to transfer knowledge from a fully-labeled source domain to an entirely unlabeled target domain. During the active learning period, some pixels in the target domain are selected and annotated as active labels through several selection rounds. Such active labels can improve the target domain model performance greatly. However, existing approaches solely rely on pseudo labels, highly-confident classifier predictions on target images, to train the initial target domain model, resulting in a sub-optimal solution for model training. This initial model will be used for active label selection. Meanwhile, previous methods use entropy-based measurement to select pixels for annotation, which fails to detect high-confidence errors in earlier selection rounds due to the absence of target information. To address these issues, we propose a prototype-guided pseudo-label generating approach that leverages the relationships between source prototypes and target features. It generates target pseudo labels based on diverse source prototypes, thereby alleviating the issue of classifier predictions. Furthermore, perturbation-based uncertainty measurement, calculating the discrepancy between the target image and the augmented one, is introduced to find the areas with unstable predictions. Extensive experiments demonstrate that our approach outperforms state-of-the-art active domain adaptation methods on two benchmarks, GTAV → Cityscapes, and SYNTHIA → Cityscapes. Comparable performance is also achieved when compared to fully-supervised methods. [Display omitted] • A prototype-guided pseudo-label generation approach is suggested to maximize the utilization of varied source domain prototypes, thereby mitigating classifier prediction bias. • The perturbation-based uncertainty measurement is introduced to measure the prediction perturbation between the original target images and augmented ones. • A balanced schedule is designed to ensure that the perturbation plays a significant role in earlier selection rounds while later rounds are more driven by entropy uncertainty. [ABSTRACT FROM AUTHOR]

3D point clouds are a crucial type of data collected by LiDAR sensors and widely used in transportation applications due to its concise descriptions and accurate localization. Deep neural networks (DNNs) have achieved remarkable success in processing large amount of disordered and sparse 3D point clouds, especially in various computer vision tasks, such as pedestrian detection and vehicle recognition. Among all the learning paradigms, Self-Supervised Learning (SSL), an unsupervised training paradigm that mines effective information from the data itself, is considered as an essential solution to solve the time-consuming and labor-intensive data labeling problems via smart pre-training task design. This paper provides a comprehensive survey of recent advances on SSL for point clouds. We first present an innovative taxonomy, categorizing the existing SSL methods into four broad categories based on the pretexts' characteristics. Under each category, we then further categorize the methods into more fine-grained groups and summarize the strength and limitations of the representative methods. We also compare the performance of the notable SSL methods in literature on multiple downstream tasks on benchmark datasets both quantitatively and qualitatively. Finally, we propose a number of future research directions based on the identified limitations of existing SSL research on point clouds. [ABSTRACT FROM AUTHOR]

• Adapt to dynamic changes in search results and carry out visual disambiguation. • A saliency-guided deep MIL network to remove outliers and learn visual disambiguation. • Identify appropriate visual senses for sense-specific image collection. • Be used as a pre-step before learning from the web for improving the performance. Due to the high cost of manual annotation, learning directly from the web has attracted broad attention. One issue that limits the performance of current webly supervised models is the problem of visual polysemy. In this work, we present a novel framework that resolves visual polysemy by dynamically matching candidate text queries with retrieved images. Specifically, our proposed framework includes three major steps: we first discover and then dynamically select the text queries according to the keyword-based image search results, we employ the proposed saliency-guided deep multi-instance learning (MIL) network to remove outliers and learn classification models for visual disambiguation. Compared to existing methods, our proposed approach can figure out the right visual senses, adapt to dynamic changes in the search results, remove outliers, and jointly learn the classification models. Extensive experiments and ablation studies on CMU-Poly-30 and MIT-ISD datasets demonstrate the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

Currently image aesthetic estimation using deep learning has achieved great success compared with the traditional methods by hand-crafted features. Similar to recognition problem, aesthetic estimation categorizes images into visually appealing or not. Nevertheless, it is desirable to understand why certain images are visually more appealing, in specific, which part of the image is contributing to the aesthetic preference. In fact, most traditional approaches adopting hand-crafted feature are, to some extent, able to understand part of image’s aesthetic and content information while few studies have been conducted in the context of deep learning. Moreover, we discover that aesthetic rating is ambiguous so that many examples are uncertain in aesthetic level. This has caused a highly imbalanced distribution of aesthetic ratings. To tackle all these issues, we propose an end-to-end convolutional neural network (CNN) model which simultaneously implements aesthetic classification and understanding. To overcome the imbalanced aesthetic ratings, a sample-specific classification method that re-weights samples’ importance is proposed. We find that dropping out ambiguous image, as common adopted by recent deep learning models, is a special case of the sample-specific method, and also figure out that as the weights of the non-ambiguous images increase, the performance is positively affected. In order to understand what is learned in the deep model, global average pooling (GAP) following the last feature map is employed to generate aesthetic activation map (AesAM) and attribute activation map (AttAM). AesAM and AttAM respectively represent the likelihood of aesthetic level for spatial location, and the likelihood of different attribute information. In particular, AesAM mainly accounts for what is learned in deep model. Experiments are carried out on public aesthetic datasets and state-of-the-art performance is achieved. Thanks to the introduction of AttAM, the aesthetic preference is explainable by visualization. Finally, a simple application on image cropping based on the AesAM is presented. The code and trained model will be publicly available on https://github.com/galoiszhang/AWCU . [ABSTRACT FROM AUTHOR]

• We present a multi-stage residual feature pyramid gating strategy which designs soft gates to select and reinforce multi-scale semantic features. • We introduce an image pyramid gating strategy to preserve natural information to enhance network discrimination in an attention-based fashion. • Our method attains the best performance without pre-training and can be readily combined with general frameworks without structure modifications. Specifically, the prediction accuracy consistently achieves over 3% improvements than the baseline (without pre-training) under various model configurations. In this work, we propose a comprehensive aggregated residual gating structure, the Pyramid GAting Network (PGA-Net) for human pose estimation which can select, distill, and fuse semantic level and natural level information from multiple scales. In comparison, through utilizing multi-scale features, most existing state-of-the-art pose estimation methods are still limited in three aspects. First, multi-scale features contain massively redundant information, which is unfortunately not distilled by most existing approaches. Second, preferring deeper network structures to extract strong semantic features, the conventional methods often ignore original texture information fusion. Third, to attain a good parameter initialization, the current methods heavily rely on pre-training, which is very time-consuming or even unavailable. While better coping with the above problems, our proposed PGA-Net distills high-level semantic features and replenishes low-level original information to reinforce module representation capability. Meanwhile, PGA-Net demonstrates notable training stability and superior performance even without pre-training. Extensive experiments demonstrate that our method consistently outperforms previous approaches even without pre-training, enabling thus an end-to-end model training from scratch. In COCO benchmark, PGA-Net consistently achieves over 3% improvements than the baseline (without pre-training) under various model configurations. 1 1 The code is released at https://github.com/ssr0512/PGA-Net [ABSTRACT FROM AUTHOR]

Long Term Evolution (LTE) network is widely used in video transmission because of its high-speed communication capacity. In this paper, an end-to-end distortion-based bandwidth allocation method for multiusers is proposed to enhance video transmission performance of the LTE network. For an LTE network, since throughput is observed as an independent variable which can affect the packet loss ratio, and thus affect the transmission distortion, the end-to-end distortion model is first derived by taking throughput into account. Then, based on the derived end-to-end distortion model, the bandwidth allocation problem is formulated as a convex optimization problem and solved by Karush–Kuhn–Tucker conditions. Simulation results demonstrate the effectiveness of the proposed method. The rate distortion performance of the proposed method is better than the average bandwidth allocation method under different bandwidth utilizations, and is very close to the exhaustive search-based method. [ABSTRACT FROM PUBLISHER]

Deep learning techniques have been extensively investigated for the purpose of further increasing the efficiency of traditional video compression. Some deep learning techniques for down/up-sampling-based video coding were found to be especially effective when the bandwidth or storage is limited. Existing works mainly differ in the super-resolution models used. Some works simply use a single image super-resolution model, ignoring the rich information in the correlation between video frames, while others explore the correlation between frames by simply concatenating the features across adjacent frames. This, however, may fail when the textures are not well aligned. In this paper, we propose to utilize neural texture transfer which exploits the semantic correlation between frames and is able to explore the correlated information even when the textures are not aligned. Meanwhile, an adaptive group of pictures (GOP) method is proposed to automatically decide whether a frame should be down-sampled or not. Experimental results show that the proposed method outperforms the standard HEVC and state-of-the-art methods under different compression configurations. When compared to standard HEVC, the BD-rate (PSNR) and BD-rate (SSIM) of the proposed method are up to -19.1% and -26.5%, respectively. • We introduce reference-based SR in down/up-sampling-based video coding method, where target and reference images are not required to be texture-aligned as required in existing methods. • We proposed an adaptive group of pictures (GOP) method to automatically decide the adaptive sampling scheme. • The neural texture transfer model for reference-based SR produces realistic up-sampled frame at the decoding end. [ABSTRACT FROM AUTHOR]

A 3-D multiview video gives users an experience that is different from that provided by a traditional video; however, it puts a huge burden on limited bandwidth resources. Mixed-resolution video in a multiview system can alleviate this problem by using different video resolutions for different views. However, to reduce visual uncomfortableness and to make this video format more suitable for free-viewpoint television, the low-resolution (LR) views need to be super-resolved to the target full resolution. In this paper, we propose a virtual-view-assisted super-resolution algorithm, where the inter-view similarity is used to determine whether the missing pixels in the super-resolved frame need to be filled by virtual-view pixels or by spatial interpolated pixels. The decision mechanism is steered by the texture characteristics of the neighbors of each missing pixel. Furthermore, the inter-view similarity is used, on the one hand, to enhance the quality of the virtual-view-copied pixels by compensating the luminance difference between different views and, on the other hand, to enhance the original LR pixels in the super-resolved frame by reducing their compression distortion. Thus, the proposed method can recover the details in regions with edges while maintaining good quality at smooth areas by properly exploiting the high-quality virtual-view pixels and the directional correlation of pixels. The experimental results demonstrate the effectiveness of the proposed approach with a peak signal-to-noise ratio gain of up to 3.85 dB. [ABSTRACT FROM AUTHOR]

Due to the prediction structures employed in video coding, the loss of one packet will affect many following frames. In this paper, a multiple description coding scheme with stagger frame order is proposed for stereoscopic 3-D videos. First, the reference and auxiliary views in stereoscopic sequences will be asymmetrically encoded into one description, whereas the other description will be formed in the same way with one dumb frame delay. Because of the stagger frame order, the coarsely encoded B frames will be inserted into different positions of the two descriptions. If a certain frame encoded with I/P mode is lost, then its corresponding B-frame version will be employed to compensate for the loss. In each description, the quantization steps of B frames are tuned based on a closed-form solution that considers the video contents, network status, frame positions in the group of picture, and the layer of the views. For further improvement, a fusing scheme is provided. The experimental results demonstrate that the proposed scheme outperforms state-of-the-art schemes. Specifically, up to 1.3-dB gain is achieved in the case of packet loss, and 2-dB gain is obtained for the side/central performance. [ABSTRACT FROM PUBLISHER]

Coding artifacts are annoying in highly compressed signals. Most of the existing artifact reduction methods are designed for one specific type of artifacts, codecs, and bitrates, which are complex and exclusive for one type of artifact reduction. Since both the compressed image/video and the coding error contain information of the original signal, they are highly correlated. Therefore, we try to recover some lost data based on the correlation between the compressed signal and the coding error, and introduce a novel and universal artifact reduction method. Firstly, according to the spatial correlation among pixels, a pixel-adaptive anisotropic filter is designed to reconstruct the distorted signal. Next, a globally optimal filter is designed to further recover the coding loss. Experimental results demonstrate that within an extensive range of bitrates, the proposed method achieves about 0.8 dB, 0.45 dB, 0.3 dB, and 0.2 dB on average of PSNR improvement for JPEG, MPEG4, H.264/AVC, and HEVC compressed signals, respectively. [ABSTRACT FROM AUTHOR]

The depth-image-based-rendering is a key technique to realize free viewpoint television. However, one critical problem in these systems is filling the disocclusion due to the 3-D warping process. This paper exploits the temporal correlation of texture and depth information to generate a background reference image. This is then used to fill the holes associated with the dynamic parts of the scene, whereas for static parts the traditional inpainting method is used. To generate the background reference image, the Gaussian mixture model is employed on the texture information, whereas, depth maps information are used to detect moving objects so as to enhance the background reference image. The proposed holes filling approach is particularly useful for the single-view-plus-depth format, where, contrary to the multi-view-plus-depth format, only information of one view could be used for this task. The experimental results show that objective and subjective gains can be achieved, and the gain ranges from 1 to 3 dB over the inpainting method. [ABSTRACT FROM PUBLISHER]

• cylinder primitive extraction is addressed as a clustering problem of possible cylinder candidates; • candidates are generated by cutting the scene through several random planes; • a novel cylinder similarity function based on dual quaternion algebra is defined; • the clustering process, based on Game Theory, inherently discards incompatible candidates; • the proposed method can deal with noisy, partially occluded and non-oriented point clouds. Generative Adversarial Networks (GAN) receive great attention recently due to its excellent performance in image generation, transformation, and super-resolution. However, less emphasis or study has been put on GAN for classification with super-resolution. Moreover, though GANs may fabricate images which perceptually looks realistic, they usually fabricate some fake details especially in character data; this would impose further difficulties when they are input for classification. In this paper, we propose a novel Generative Adversarial Classifier (GAC) for low-resolution handwriting character recognition. Specifically, we design an additional classifier component in GAC, leading to a novel three-player GAN model which is not only able to generate high-quality super-resolved images, but also favorable for classification. Experimental results show that our proposed method can obtain remarkable performance in handwriting characters with 8 × super-resolution, achieving new state-of-the-art on benchmark dataset CASIA-HWDB1.1, and MNIST. [ABSTRACT FROM AUTHOR]

• We propose a novel geometry based algorithm for accurately estimating the head pose from a single 2D face image at a very low computational cost. Specifically, the rectangular coordinates of only four non-coplanar feature points from a predefined 3D facial model as well as the corresponding ones automatically/manually extracted from a 2D face image are first normalized to exclude the effect of external factors (i.e., scale factor and translation parameters). Then, the four normalized 3D feature points are represented in spherical coordinates with reference to the uniquely determined sphere by themselves, which are further iteratively refined via 3D morphing. Due to the spherical parameterization, the coordinates of feature points could be morphed along all the three directions effectively. Finally, the rotation matrix indicating the head pose is obtained by minimizing the Euclidean distance between the normalized 2D feature points and the 2D re-projections of morphed 3D feature points. Head pose estimation plays a vital role in various applications, e.g., driver-assistance systems, human-computer interaction, virtual reality technology, and so on. We propose a novel geometry-based method for accurately estimating the head pose from a single 2D face image at a very low computational cost. Specifically, the rectangular coordinates of only four non-coplanar feature points from a predefined 3D facial model as well as the corresponding ones automatically/manually extracted from a 2D face image are first normalized to exclude the effect of external factors (i.e., scale factor and translation parameters). Then, the four normalized 3D feature points are represented in spherical coordinates with reference to the uniquely determined sphere by themselves. Due to the spherical parametrization, the coordinates of feature points can then be morphed along all the three directions in the rectangular coordinates effectively. Finally, the rotation matrix indicating the head pose is obtained by minimizing the Euclidean distance between the normalized 2D feature points and the 2D re-projections of the morphed 3D feature points. Comprehensive experimental results over two popular datasets, i.e., Pointing'04 and Biwi Kinect , demonstrate that the proposed method can estimate head poses with higher accuracy and lower run time than state-of-the-art geometry-based methods. Even compared with start-of-the-art learning-based methods or geometry-based methods with additional depth information, our method still produces comparable performance. [ABSTRACT FROM AUTHOR]