Search

Weakly supervised object localization (WSOL) tasks aim to classify and locate a single object under the supervision of only image-level labels. Pseudo-supervised learning methods have been shown to be effective for WSOL. These methods divide WSOL tasks into two decoupled subtasks: classification and localization. The decoupled framework has been proven to be effective in improving the performance of the localization subtask, but the predicted localizations are not robust enough due to the noise of pseudo-labels. Based on the assumption that the localization model should make similar predictions on different versions of the same image, we propose an additional refinement stage to learn more consistent localization. In the refinement stage, we propose a simple and effective method for evaluating the confidence of pseudo-labels based on classification discrimination. By learning consistency from high-confidence pseudo-labels, we further refine the localization model to obtain more consistent and more accurate predictions. Additionally, in the initialization stage, we propose a mask-based pseudo-label generator to initialize the localization model. We conducted experiments on two benchmark datasets: CUB-200-2011 and ImageNet-1k. The experimental results show that our two-stage approach achieved 94.01% GT-Konwn localization accuracy on the CUB-200-2011 testing dataset and 65.23% GT-Konwn localization accuracy on the ImageNet-1k validation dataset. Moreover, when directly applied to the pseudo-supervised localization model, our refinement stage achieved 94.05% and 67.13% GT-Konwn localization accuracy on the CUB-200-2011 and ImageNet-1k datasets, respectively, which outperforms the corresponding pseudo-supervised localization model with an accuracy improvement of 3.34% and 2.34%, respectively.

In visual surveillance, deep learning-based foreground object detection algorithms are superior to classical background subtraction (BGS)-based algorithms. However, deep learning-based methods are limited because detection performance deteriorates in a new environment different from the training environment. This limitation can be solved by retraining the model using additional ground-truth labels in the new environment. However, generating ground-truth labels for visual surveillance is time-consuming and expensive. This paper proposes a method that does not require foreground labels when adapting to a new environment. To this end, we propose an integrated network that produces two kinds of outputs a background model image and a foreground object map. We can adapt to the new environment by retraining using a background model image. The proposed method consists of one encoder and two decoders for detecting foreground objects and a background model image. It is designed to enable real-time processing with desktop GPUs. The proposed method shows 14.46% improved FM in a new environment different from training and 11.49% higher FM than the latest BGS algorithm.

Weakly supervised semantic segmentation (WSSS) methods have received significant attention in recent years, since they can dramatically reduce the annotation costs of fully supervised alternatives. While most previous studies focused on leveraging classification labels, we explore instead the use of image captions, which can be obtained easily from the web and contain richer visual information. Existing methods for this task assigned text snippets to relevant semantic labels by simply matching class names, and then employed a model trained to localize arbitrary text in images to generate pseudo-ground truth segmentation masks. Instead, we propose a dedicated caption processing module to extract structured supervision from captions, consisting of improved relevant object labels, their visual attributes, and additional background categories, all of which are useful for improving segmentation quality. This module uses syntactic structures learned from text data, and semantic relations retrieved from a knowledge database, without requiring additional annotations on the specific image domain, and consequently can be extended immediately to new object categories. We then present a novel localization network, which is trained to localize only these structured labels. This strategy simplifies model design, while focusing training signals on relevant visual information. Finally, we describe a method for leveraging all types of localization maps to obtain high-quality segmentation masks, which are used to train a supervised model. On the challenging MS-COCO dataset, our method moves the state-of-the-art forward significantly for WSSS with image-level supervision by a margin of 7.6% absolute (26.7% relative) mean Intersection-over-Union, achieving 54.5% precision and 50.9% recall.

Weakly supervised instance segmentation, which could greatly decrease financial and time cost, is one of fundamental computer vision tasks. State-of-the-art methods mainly concentrate on improving the quality of generated pixel level labels, namely masks, using complex traditional segmentation methods, and ignore the effect of the quality of generated masks. Namely, the masks of small object instances tend to be invalid, which would degrades the performance of instance segmentation. In this paper, we propose a two-stage transfer learning framework for weakly supervised instance segmentation. We explicitly discriminate the invalid and valid generated masks, and just utilize the valid masks for training to avoid the interference of invalid ones. We use a network-based transfer learning strategy to effectively utilize all useful information, including category labels and bounding-box information of all objects and valid generated masks. Besides, we further use a feature-mapping-based transfer learning strategy to improve the performance of small object instance segmentation. We demonstrate the effectiveness of the proposed method on the PASCAL VOC 2012, and the experimental results show that our proposed method is effective and outperforms state-of-the-art methods.

Most temporal action localization methods are usually trained using video data-sets with frame-wise annotations which are expensive and time-consuming to acquire. To alleviate this problem, many weakly supervised temporal action localization methods which only leverage video-level annotations during training are proposed. In this paper, we first analyze three problems of weakly supervised temporal action localization, namely feature similarity, action completeness, and weak annotation. Based on these three problems, we propose a novel network called multi-stage fusion network, which decomposes the problems into three different modules within the network, namely feature, sub-action, and action modules. Specifically, for feature similarity, a Triplet Loss was introduced to ensure the action instances from the same class having similar feature sequences and expand the margin of the action instance from different classes in the feature module. For action completeness, each stage of the sub-action module can discover the different sub-actions. The complete action instances can be localized in the action module by fusing multiple sub-actions from the sub-action module. To alleviate weak annotation, we localize multiple action proposals from multi-stage outputs of the network in the action module and select the action proposals with higher confidence scores as predicted action instances. Extensive experiment results on data-sets Thumos'14 and ActivityNet1.2 demonstrate that our method outperforms the state-of-the-art methods and the average mean Average Precision (mAP) on Thumos'14 is significantly improved from 40.9% to 43.3%.

Mixed dish, which mixes different types of dishes in one plate, is a popular kind of food in East and Southeast Asia. Identifying the dish type in the mixed dish is essential for dietary tracking, which gains increasing research attention recently. Nevertheless, mixed dish detection is a challenging task because of large visual variances among dishes in different canteens, which is known as the domain shifting problem. Since collecting and annotating sufficient training samples in each canteen for model training is difficult, a more practical way is developing detection models that can adapt quickly to cross-canteen mixed-dish detection with less supervision information. To this end, we propose a novel framework called Weakly-supervised Mean Teacher Network (WMT-Net) that addresses this specific detection task in a weakly supervised manner, where bounding box annotations are not required in the target domain. The proposed WMT-Net constructs Mean Teacher learning by maintaining the image-level consistency between teacher and student modules. Specifically, WMT-Net firstly learns instance-level information from the source dataset in a fully supervised fashion for the student model. Then the whole architecture is optimized with weakly supervised learning: 1) weakly supervised training in student model to reduce the domain gap in global semantics between source data and target data, 2) image-level consistency to align the image-level predictions between teacher model and student model. Experimental results on mixed-dish dataset show that even the proposed WMT-Net is trained in a weakly supervised fashion on the target domain, the performances attained by WMT-Net are very close to the model trained in a fully supervised fashion, which verify the effectiveness of WMT-Net. In addition, the proposed WMT-Net also achieves 44.6% mAP on Pascal VOC to Clipart cross-domain detection, which improves 7.2% mAP compared with the state-of-the-arts method and further demonstrates its generalization capabilities.

Human activity recognition from the video is an important problem due to its potential applications in remote surveillance, content-based video retrieval, and in humanoid robots. Most of the visual activity recognition research in the last decade focus on recognizing basic human actions from well-constrained laboratory videos and depend upon fully annotated video dataset for model training. Whereas activity recognition from unconstrained videos is still very challenging due to large variations in object appearance and pose, occlusion, and inter and intraclass variations. It is an extremely laborious task to prepare large scale realistic video activity dataset with detailed annotations of the human, object, and their mutual interactions in each frame. Although it is intuitive to model contextual relationships form fully annotated dataset, however, it is unknown as to how reliably multilevel contextual features can be extracted in the absence of annotated dataset. To mitigate these challenges, we propose a weakly supervised approach for complex human activity recognition from realistic videos. The proposed approach requires only activity labels for each video to train the model. A novel multilevel contextual features and context estimation procedure from the un-annotated dataset is also introduced. Restricted Boltzman machine is used to systematically integrate multilevel contextual features. We evaluate the proposed approach on benchmark realistic surveillance video datasets for human-human and human-object interaction activity recognition. The experimental results show improved accuracies on benchmark datasets without using fully annotated datasets.

Deep convolutional neural networks (DCNNs) trained on the pixel-wise annotated images have dramatically improved the state-of-the-art in semantic segmentation. However, due to the high cost of labeling training data, its application has great limitation. In this paper, we propose a DCNNs model for generating the pixel-level labels using the image-level annotation. The model consists of an encoder-decoder, a feature decomposer, and a multi-label classifier. The encoder extracts the deep convolutional feature maps of the input image. The feature decomposer can decompose the convolutional feature extracted by the encoder into feature components of different semantics. The decomposer is based on the orthogonal non-negative matrix factorization (NMF) technology. The function of the decoder is to map the feature components of different semantics to the input resolution images. The decoder uses the position index of maximum pooling provided by the corresponding encoder to perform non-linear up-sampling, which eliminates the learning requirement of up-sampling. Since the image reconstruction is conducted according to the semantic categories, image regions of different semantics are restored to different images. Then, the regions of different semantics can be segmented through the posted-processing algorithms. The experimental results on open data set show that the proposed model outperforms some recently developed methods.

Image semantic segmentation has great development in many fields, and the lack of fully supervised segmentation labels has always been a major problem in the development of image semantic segmentation. In this paper, we propose a WAILS method to solve this problem. First, the image is coarsely segmented through a weakly supervised network at the image level. Second, to further obtain the shape of the target in the image, the watershed algorithm is used to refine the result of the coarse segmentation. Third, this refined image is used as a label for the first round of training of a fully supervised image semantic segmentation network. At last, the results are refined through the watershed as the label of the second round of fully supervised training and then it iterates. This method achieves the pixel-level semantic segmentation only through image-level labels and watershed pre-segmentation. Our method achieved good performance on the PASCAL 2012 dataset and the COCO dataset, while its segmentation accuracy surpasses all current weakly supervised semantic segmentation models in the category of bird, airplane, sheep, and so on.