Search

Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task. In this work, we aim to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue due to data scarcity, which is a critical problem in low-shot regimes. To this end, we present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of two key components: a video-text contrastive objective and a prototype modulation. Specifically, the former bridges the task discrepancy between CLIP and the few-shot video task by contrasting videos and corresponding class text descriptions. The latter leverages the transferable textual concepts from CLIP to adaptively refine visual prototypes with a temporal Transformer. By this means, CLIP-FSAR can take full advantage of the rich semantic priors in CLIP to obtain reliable prototypes and achieve accurate few-shot classification. Extensive experiments on five commonly used benchmarks demonstrate the effectiveness of our proposed method, and CLIP-FSAR significantly outperforms existing state-of-the-art methods under various settings. The source code and models are publicly available at https://github.com/alibaba-mmai-research/CLIP-FSAR. [ABSTRACT FROM AUTHOR]

Generative Adversarial Networks (GANs) can synthesize more and more realistic images. However, one fundamental issue hinders their practical applications: the incapability of encoding real samples in the latent space. Many semantic image editing applications rely on inverting the given image into the latent space and then manipulating inverted code. One possible solution is to learn an encoder for GAN via Variational Auto-Encoder. However, the entanglement of the latent space poses a major challenge for learning the encoder. To tackle the challenge and enable inference in GANs, we propose a novel method named Latently Invertible Autoencoder (LIA). In LIA, an invertible network and its inverse mapping are symmetrically embedded in the latent space of an autoencoder. The decoder of LIA is first trained as a standard GAN with the invertible network, and then the encoder is learned from a disentangled autoencoder by detaching the invertible network from LIA. It thus avoids the entanglement problem caused by the latent space. Extensive experiments on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA for the image inversion and its applications. Code and models are available at https://github.com/genforce/lia. [ABSTRACT FROM AUTHOR]

Existing action recognition algorithms require a set of positive exemplars to train a classifier for each action. However, the amount of action classes is very large and the users' queries vary dramatically. It is impractical to pre-define all possible action classes beforehand. To address this issue, we propose to perform action recognition with no positive exemplars, which is often known as the zero-shot learning. Current zero-shot learning paradigms usually train a series of attribute classifiers and then recognize the target actions based on the attribute representation. To ensure the maximum coverage of ad-hoc action classes, the attribute-based approaches require large numbers of reliable and accurate attribute classifiers, which are often unavailable in the real world. In this paper, we propose an approach that merely takes an action name as the input to recognize the action of interest without any pre-trained attribute classifiers and positive exemplars. Given an action name, we first build an analogy pool according to an external ontology, and each action in the analogy pool is related to the target action at different levels. The correlation information inferred from the external ontology may be noisy. We then propose an algorithm, namely adaptive multi-model rank-preserving mapping (AMRM), to train a classifier for action recognition, which is able to evaluate the relatedness of each video in the analogy pool adaptively. As multiple mapping models are employed, our algorithm has better capability to bridge the gap between visual features and the semantic information inferred from the ontology. Extensive experiments demonstrate that our method achieves the promising performance for action recognition only using action names, while no attributes and positive exemplars are available. [ABSTRACT FROM AUTHOR]