Search

The visual security index (VSI) is a quantized indicator for objective visual security evaluation of selectively encrypted images. One challenging problem in current research is that the performance of VSIs is highly sensitive to the extracted features and the method of similarity measurement, and it is hard to choose appropriate handcrafted features from encrypted images, as well as to find an effective similarity measurement. In this paper, we make the first attempt to present a novel convolutional neural network-based visual security index (CNNVSI). Our proposed CNNVSI is purely data-driven and trained end-to-end. We propose three specialized designs to make the approach work for encrypted low-quality images without any handcrafted features or prior knowledge about the human vision system (HVS). First, we present a patch labeling algorithm to assign each encrypted patch a visual security score. Second, we design a multiscale attention residual network (MARNet) for feature learning. Last, we propose to fuse the learned features from plain images, encrypted images and their discrepancy images. Extensive and systematic experiments are conducted on five publicly available image databases to analyze the performance of our proposed CNNVSI, and the experimental results and their analysis demonstrate that our proposed CNNVSI significantly outperforms the existing state-of-the-art methods in terms of accuracy and stability. [ABSTRACT FROM AUTHOR]

Perceptual/selective encryption has been gaining widespread attention as an emerging technology for image privacy protection. However, few studies focus on the visual security evaluation of perceptually encrypted images, which has a significant impact on measuring the effectiveness and practicality of these encryption methods. In this paper, we propose an image importance-based visual security index (IIBVSI) by leveraging spatial contrast and texture features. Based on the characteristics of perceptually encrypted images, we present an averaged high-order gradient magnitude map to describe the spatial contrast feature and introduce a combined local amplitude map of multiple log-Gabor filters to represent the texture feature. Specifically, the multiresolution representation of an image is first created by downsampling to simulate the hierarchical property of the human visual system. Next, for each scale of image resolution, the spatial contrast and the texture feature maps are extracted from both plain and encrypted images. Similarity measurements are then conducted on these feature maps to generate the contrast and the texture similarity maps. An image importance-based pooling strategy is subsequently proposed to combine these measurements and generate a visual security score. The final IIBVSI score is computed by averaging the visual security scores of all scales of image resolution. Extensive experiments are conducted on several publicly available databases, and the results demonstrate the superiority and robustness of our proposed IIBVSI compared with existing state-of-the-art work in the low and moderate image quality ranges. [ABSTRACT FROM AUTHOR]

Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model overfitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis. [ABSTRACT FROM AUTHOR]

Decentralized learning has gained great popularity to improve learning efficiency and preserve data privacy. Each computing node makes equal contribution to collaboratively learn a Deep Learning model. The elimination of centralized Parameter Servers (PS) can effectively address many issues such as privacy, performance bottleneck and single-point-failure. However, how to achieve Byzantine Fault Tolerance in decentralized learning systems is rarely explored, although this problem has been extensively studied in centralized systems. In this paper, we present an in-depth study towards the Byzantine resilience of decentralized learning systems with two contributions. First, from the adversarial perspective, we theoretically illustrate that Byzantine attacks are more dangerous and feasible in decentralized learning systems: even one malicious participant can arbitrarily alter the models of other participants by sending carefully crafted updates to its neighbors. Second, from the defense perspective, we propose Ubar, a novel algorithm to enhance decentralized learning with Byzantine Fault Tolerance. Specifically, Ubar provides a Uniform Byzantine-resilient Aggregation Rule for benign nodes to select the useful parameter updates and filter out the malicious ones in each training iteration. It guarantees that each benign node in a decentralized system can train a correct model under very strong Byzantine attacks with an arbitrary number of faulty nodes. We conduct extensive experiments on standard image classification tasks and the results indicate that Ubar can effectively defeat both simple and sophisticated Byzantine attacks with higher performance efficiency than existing solutions. [ABSTRACT FROM AUTHOR]

Image classification is a fundamental artificial intelligence task that labels images into one of some predefined classes. However, training complex image classification models requires a large amount of computation resources and data in order to reach state-of-the-art performance. This demand drives the growth of distributed deep learning, where multiple agents cooperatively train global models with their individual datasets. Among such learning systems, decentralized learning is particularly attractive, as it can improve the efficiency and fault tolerance by eliminating the centralized parameter server, which could be the single point of failure or performance bottleneck. Although the agents do not need to disclose their training image samples, they exchange parameters with each other at each iteration, which can put them at the risk of data privacy leakage. Past works demonstrated the possibility of recovering training images from the exchanged parameters. One common defense direction is to adopt Differential Privacy (DP) to secure the optimization algorithms such as Stochastic Gradient Descent (SGD). Those DP-based methods mainly focus on standalone systems, or centralized distributed learning. How to enforce and optimize DP protection in decentralized learning systems is unknown and challenging, due to their complex communication topologies and distinct learning characteristics. In this paper, we design TOP- DP, a novel solution to optimize the differential privacy protection of decentralized image classification systems. The key insight of our solution is to leverage the unique features of decentralized communication topologies to reduce the noise scale and improve the model usability. (1) We enhance the DP-SGD algorithm with this topology-aware noise reduction strategy, and integrate the time-aware noise decay technique. (2) We design two novel learning protocols (synchronous and asynchronous) to protect systems with different network connectivities and topologies. We formally analyze and prove the DP requirement of our proposed solutions. Experimental evaluations demonstrate that our solution achieves a better trade-off between usability and privacy than prior works. To the best of our knowledge, this is the first DP optimization work from the perspective of network topologies. [ABSTRACT FROM AUTHOR]

In this paper, we focus on learning semantic representations for large-scale highly abstract sketches that were produced by the practical sketch-based application rather than the excessively well dawn sketches obtained by crowd-sourcing. We propose a dual-branch CNN-RNN network architecture to represent sketches, which simultaneously encodes both the static and temporal patterns of sketch strokes. Based on this architecture, we further explore learning the sketch-oriented semantic representations in two practical settings, i.e., hashing retrieval and zero-shot recognition on million-scale highly abstract sketches produced by practical online interactions. Specifically, we use our dual-branch architecture as a universal representation framework to design two sketch-specific deep models: (i) We propose a deep hashing model for sketch retrieval, where a novel hashing loss is specifically designed to further accommodate both the abstract and messy traits of sketches. (ii) We propose a deep embedding model for sketch zero-shot recognition, via collecting a large-scale edge-map dataset and proposing to extract a set of semantic vectors from edge-maps as the semantic knowledge for sketch zero-shot domain alignment. Both deep models are evaluated by comprehensive experiments on million-scale abstract sketches produced by a global online game QuickDraw and outperform state-of-the-art competitors. [ABSTRACT FROM AUTHOR]

Human action can be recognized from a single still image by modeling human–object interactions (HOIs), which infers the mutual spatial structure information between human and the manipulated object as well as their appearance. Existing approaches rely heavily on accurate detection of human and object and estimation of human pose; they are thus sensitive to large variations of human poses, occlusion, and unsatisfactory detection of small size objects. To overcome this limitation, a novel exemplar-based approach is proposed in this paper. Our approach learns a set of spatial pose–object interaction exemplars, which are probabilistic density functions describing spatially how a person is interacting with a manipulated object for different activities. Specifically, a new framework consisting of an exemplar-based HOI descriptor and an associated matching model is formulated for robust human action recognition in still images. In addition, the framework is extended to perform HOI recognition in videos, where the proposed exemplar representation is used for implicit frame selection to negate irrelevant or noisy frames by temporal structured HOI modeling. Extensive experiments are carried out on two image action datasets and two video action datasets. The results demonstrate the effectiveness of our proposed methods and show that our approach is able to achieve state-of-the-art performance, compared with several recently proposed competitors. [ABSTRACT FROM PUBLISHER]