Search

Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task. In this work, we aim to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue due to data scarcity, which is a critical problem in low-shot regimes. To this end, we present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of two key components: a video-text contrastive objective and a prototype modulation. Specifically, the former bridges the task discrepancy between CLIP and the few-shot video task by contrasting videos and corresponding class text descriptions. The latter leverages the transferable textual concepts from CLIP to adaptively refine visual prototypes with a temporal Transformer. By this means, CLIP-FSAR can take full advantage of the rich semantic priors in CLIP to obtain reliable prototypes and achieve accurate few-shot classification. Extensive experiments on five commonly used benchmarks demonstrate the effectiveness of our proposed method, and CLIP-FSAR significantly outperforms existing state-of-the-art methods under various settings. The source code and models are publicly available at https://github.com/alibaba-mmai-research/CLIP-FSAR. [ABSTRACT FROM AUTHOR]

Generative Adversarial Networks (GANs) can synthesize more and more realistic images. However, one fundamental issue hinders their practical applications: the incapability of encoding real samples in the latent space. Many semantic image editing applications rely on inverting the given image into the latent space and then manipulating inverted code. One possible solution is to learn an encoder for GAN via Variational Auto-Encoder. However, the entanglement of the latent space poses a major challenge for learning the encoder. To tackle the challenge and enable inference in GANs, we propose a novel method named Latently Invertible Autoencoder (LIA). In LIA, an invertible network and its inverse mapping are symmetrically embedded in the latent space of an autoencoder. The decoder of LIA is first trained as a standard GAN with the invertible network, and then the encoder is learned from a disentangled autoencoder by detaching the invertible network from LIA. It thus avoids the entanglement problem caused by the latent space. Extensive experiments on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA for the image inversion and its applications. Code and models are available at https://github.com/genforce/lia. [ABSTRACT FROM AUTHOR]