1. A Generative Model for Texture Synthesis based on Optimal Transport between Feature Distributions
- Author
-
Houdard, Antoine, Leclaire, Arthur, Papadakis, Nicolas, Rabin, Julien, Institut de Mathématiques de Bordeaux (IMB), Université Bordeaux Segalen - Bordeaux 2-Université Sciences et Technologies - Bordeaux 1-Université de Bordeaux (UB)-Institut Polytechnique de Bordeaux (Bordeaux INP)-Centre National de la Recherche Scientifique (CNRS), Ubisoft, Groupe de Recherche en Informatique, Image et Instrumentation de Caen (GREYC), Centre National de la Recherche Scientifique (CNRS)-École Nationale Supérieure d'Ingénieurs de Caen (ENSICAEN), Normandie Université (NU)-Normandie Université (NU)-Université de Caen Normandie (UNICAEN), Normandie Université (NU), Université de Caen Normandie (UNICAEN), and ANR-16-CE33-0010,GOTMI,Generalized Optimal Transport Models for Image processing(2016)
- Subjects
FOS: Computer and information sciences ,Statistics and Probability ,Computer Science - Machine Learning ,Computer Vision and Pattern Recognition (cs.CV) ,Applied Mathematics ,Computer Science - Computer Vision and Pattern Recognition ,Machine Learning (stat.ML) ,Condensed Matter Physics ,Machine Learning (cs.LG) ,[INFO.INFO-TS]Computer Science [cs]/Signal and Image Processing ,Statistics - Machine Learning ,Modeling and Simulation ,Geometry and Topology ,Computer Vision and Pattern Recognition - Abstract
We propose GOTEX, a general framework for texture synthesis by optimization that constrains the statistical distribution of local features. While our model encompasses several existing texture models, we focus on the case where the comparison between feature distributions relies on optimal transport distances. We show that the semi-dual formulation of optimal transport allows to control the distribution of various possible features, even if these features live in a high-dimensional space. We then study the resulting minimax optimization problem, which corresponds to a Wasserstein generative model, for which the inner concave maximization problem can be solved with standard stochastic gradient methods. The alternate optimization algorithm is shown to be versatile in terms of applications, features and architecture; in particular it allows to produce high-quality synthesized textures with different sets of features. We analyze the results obtained by constraining the distribution of patches or the distribution of responses to a pre-learned VGG neural network. We show that the patch representation can retrieve the desired textural aspect in a more precise manner. We also provide a detailed comparison with state-of-the-art texture synthesis methods. The GOTEX model based on patch features is also adapted to texture inpainting and texture interpolation. Finally, we show how to use our framework to learn a feed-forward neural network that can synthesize on-the-fly new textures of arbitrary size in a very fast manner. Experimental results and comparisons with the mainstream methods from the literature illustrate the relevance of the generative models learned with GOTEX.
- Published
- 2021