Knowledge-based generative adversarial networks for scene understanding in Cultural Heritage

This study shows the results of an interdisciplinary research aimed at devising artificial intelligence methods to support data understanding in Cultural Heritage. The objective is to create automated methods for automatically classifying and successively retrieving photographic data leveraging the current breed of AI methods based on Deep Learning paradigm. In this work, the lack of images to be used for the training of the AI system is addressed by testing an approach based on Generative Adversarial Networks (GANs) for automatically synthesizing unrealistic photos, which, in turn, can be used for training image classification and retrieval systems. Specifically, we propose a method to drive the generation of realistic classical order images using GAN approaches anchored to semantic ontology domain representation. More specifically, the proposed AI leverages the advantages of knowledge-based and data-driven methods and foresees a level image generation: the first one that synthesizes isolated objects corresponding to each ontology concept and the second one that, instead, combines the generated objects according to the spatial information provided by the ontology in order to generate realistic scenes. The resulting images can then be employed to train automated classifiers and more reliable retrieval methods by minimizing the efforts usually required by human operators to manually annotate data. In addition, the proposed generative strategy allows for a deeper understanding of cultural heritage visual data enabling the possibility to enhance design capabilities in simulation contexts.