Search

L'anàlisi de moviment i seguiment d'objectes ha estat un dels pricipals focus d'atenció en la comunitat de visió per computador durant les dues darreres dècades. L'interès per aquesta àrea de recerca resideix en el seu ample ventall d'aplicabilitat, que s'extén des de tasques de navegació de vehicles autònoms i robots, fins a aplications en la indústria de l'entreteniment i realitat virtual.Tot i que s'han aconseguit resultats espectaculars en problemes específics, el seguiment d'objectes continua essent un problema obert, ja que els mètodes disponibles són propensos a ser sensibles a diversos factors i condicions no estacionàries de l'entorn, com ara moviments impredictibles de l'objecte a seguir, canvis suaus o abruptes de la il·luminació, proximitat d'objectes similars o fons confusos. Enfront aquests factors de confusió la integració de múltiples característiques ha demostrat que permet millorar la robustesa dels algoritmes de seguiment. En els darrers anys, degut a la creixent capacitat de càlcul dels ordinadors, hi ha hagut un significatiu increment en el disseny de complexes sistemes de seguiment que consideren simultàniament múltiples característiques de l'objecte. No obstant, la majoria d'aquests algoritmes estan basats enheurístiques i regles ad-hoc formulades per aplications específiques, fent-ne impossible l'extrapolació a noves condicions de l'entorn.En aquesta tesi proposem un marc probabilístic general per integrar el nombre de característiques de l'objecte que siguin necessàries, permetent que interactuin mútuament per tal d'estimar-ne el seu estat amb precisió, i per tant, estimar amb precisió la posició de l'objecte que s'està seguint. Aquest marc, s'utilitza posteriorment per dissenyar un algoritme de seguiment, que es valida en diverses seqüències de vídeo que contenen canvis abruptes de posició i il·luminació, camuflament de l'objecte i deformacions no rígides. Entre les característiques que s'han utilitzat per representar l'objecte, cal destacar la paramatrització robusta del color en un espai de color dependent de l'objecte, que permet distingir-lo del fons més clarament que altres espais de color típicament ulitzats al llarg de la literatura.En la darrera part de la tesi dissenyem una tècnica per re-il·luminar tant escenes estàtiques com en moviment, de les que s'en desconeix la geometria. La re-il·luminació es realitza amb un mètode 'basat en imatges', on la generació de les images de l'escena sota noves condicions d'il·luminació s'aconsegueix a partir de combinacions lineals d'un conjunt d'imatges de referència pre-capturades, i que han estat generades il·luminant l'escena amb patrons de llum coneguts. Com que la posició i intensitat de les fonts d'il.luminació que formen aquests patrons de llum es pot controlar, és natural preguntar-nos: quina és la manera més òptima d'il·luminar una escena per tal de reduir el nombre d'imatges de referència? Demostrem que la millor manera d'il·luminar l'escena (és a dir, la que minimitza el nombre d'imatges de referència) no és utilitzant una seqüència de fonts d'il·luminació puntuals, com es fa generalment, sinó a través d'una seqüència de patrons de llum d'una base d'il·luminació depenent de l'objecte. És important destacar que quan es re-il·luminen seqüències de vídeo, les imatges successives s'han d'alinear respecte a un sistema de coordenades comú. Com que cada imatge ha estat generada per un patró de llum diferent il·uminant l'escena, es produiran canvis d'il·luminació bruscos entre imatges de referència consecutives. Sota aquestes circumstàncies, el mètode de seguiment proposat en aquesta tesi juga un paper fonamental. Finalment, presentem diversos resultats on re-il·luminem seqüències de vídeo reals d'objectes i cares d'actors en moviment. En cada cas, tot i que s'adquireix un únic vídeo, som capaços de re-il·luminar una i altra vegada, controlant la direcció de la llum, la seva intensitat, i el color., Motion analysis and object tracking has been one of the principal focus of attention over the past two decades within the computer vision community. The interest of this research area lies in its wide range of applicability, extending from autonomous vehicle and robot navigation tasks, to entertainment and virtual reality applications.Even though impressive results have been obtained in specific problems, object tracking is still an open problem, since available methods are prone to be sensitive to several artifacts and non-stationary environment conditions, such as unpredictable target movements, gradual or abrupt changes of illumination, proximity of similar objects or cluttered backgrounds. Multiple cue integration has been proved to enhance the robustness of the tracking algorithms in front of such disturbances. In recent years, due to the increasing power of the computers, there has been a significant interest in building complex tracking systems which simultaneously consider multiple cues. However, most of these algorithms are based on heuristics and ad-hoc rules formulated for specific applications, making impossible to extrapolate them to new environment conditions.In this dissertation we propose a general probabilistic framework to integrate as many object features as necessary, permitting them to mutually interact in order to obtain a precise estimation of its state, and thus, a precise estimate of the target position. This framework is utilized to design a tracking algorithm, which is validated on several video sequences involving abrupt position and illumination changes, target camouflaging and non-rigid deformations. Among the utilized features to represent the target, it is important to point out the use of a robust parameterization of the target color in an object dependent colorspace which allows to distinguish the object from the background more clearly than other colorspaces commonly used in the literature.In the last part of the dissertation, we design an approach for relighting static and moving scenes with unknown geometry. The relighting is performed through an -image-based' methodology, where the rendering under new lighting conditions is achieved by linear combinations of a set of pre-acquired reference images of the scene illuminated by known light patterns. Since the placement and brightness of the light sources composing such light patterns can be controlled, it is natural to ask: what is the optimal way to illuminate the scene to reduce the number of reference images that are needed? We show that the best way to light the scene (i.e., the way that minimizes the number of reference images) is not using a sequence of single, compact light sources as is most commonly done, but rather to use a sequence of lighting patterns as given by an object-dependent lighting basis. It is important to note that when relighting video sequences, consecutive images need to be aligned with respect to a common coordinate frame. However, since each frame is generated by a different light pattern illuminating the scene, abrupt illumination changes between consecutive reference images are produced. Under these circumstances, the tracking framework designed in this dissertation plays a central role. Finally, we present several relighting results on real video sequences of moving objects, moving faces, and scenes containing both. In each case, although a single video clip was captured, we are able to relight again and again, controlling the lighting direction, extent, and color.

Tesis doctoral presentada para lograr el título de Doctor por la Universidad Politécnica de Cataluña.--2021-07-23, [EN] Performing semantic interpretation of fashion images is undeniably one of the most challenging domains for computer vision. Subtle variations in color and shape might confer different meanings or interpretations to an image. Not only is it a domain tightly coupled with human understanding, but also with scene interpretation and context. Being able to extract fashion-specific information from images and interpret that information in a proper manner can be useful in many situations and help understanding the underlying information in an image. Fashion is also one of the most important businesses around the world, with an estimated value of 3 trillion dollars and a constantly growing online market, which increases the utility of image-based algorithms to search, classify or recommend garments. This doctoral thesis aims to solve specific problems related with the treatment of fashion e-commerce data, from low-level pure pixel information to high-level abstract conclusions of the garments appearing in an image, taking advantage of the multi-modality of the available data for developing some of the solutions. The contributions include: - A new superpixel extraction method focused on improving the annotation process for clothing images. - The construction of an image and text embedding for fashion data. - The application of this embedding space to the task of retrieving the main product in an image showing a complete outfit. In summary, fashion is a complex computer vision and machine learning problem at many levels, and developing specific algorithms that are able to capture essential information from pictures and text is not trivial. In order to solve some of the challenges it proposes, and taking into account that this is an Industrial Ph.D., we contribute with a variety of solutions that can boost the performance of many tasks useful for the fashion e-commerce industry., [ES] La interpretación semántica de imágenes del mundo de la moda es sin duda uno de los dominios más desafiantes para la visión por computador. Leves variaciones en color y forma pueden conferir significados o interpretaciones distintas a una imagen. Es un dominio estrechamente ligado a la comprensión humana subjetiva, pero también a la interpretación y reconocimiento de escenarios y contextos. Ser capaz de extraer información específica sobre moda de imágenes e interpretarla de manera correcta puede ser útil en muchas situaciones y puede ayudar a entender la información subyacente en una imagen. Además, la moda es uno de los negocios más importantes a nivel global, con un valor estimado de tres trillones de dólares y un mercado online en constante crecimiento, lo cual aumenta el interés de los algoritmos basados en imágenes para buscar, clasificar o recomendar prendas. Esta tesis doctoral pretende resolver problemas específicos relacionados con el tratamiento de datos de tiendas virtuales de moda, yendo desde la información más básica a nivel de píxel hasta un entendimiento más abstracto que permita extraer conclusiones sobre las prendas presentes en una imagen, aprovechando para ello la Multi-modalidad de los datos disponibles para desarrollar algunas de las soluciones. Las contribuciones incluyen: - Un nuevo método de extracción de superpíxeles enfocado a mejorar el proceso de anotación de imágenes de moda. - La construcción de un espacio común para representar imágenes y textos referentes a moda. - La aplicación de ese espacio en la tarea de identificar el producto principal dentro de una imagen que muestra un conjunto de prendas. En resumen, la moda es un dominio complejo a muchos niveles en términos de visión por computador y aprendizaje automático, y desarrollar algoritmos específicos capaces de capturar la información esencial a partir de imágenes y textos no es una tarea trivial. Con el fin de resolver algunos de los desafíos que esta plantea, y considerando que este es un doctorado industrial, contribuimos al tema con una variedad de soluciones que pueden mejorar el rendimiento de muchas tareas extremadamente útiles para la industria de la moda online

Trabajo presentado en la IEEE/CVF Conference on Computer Vision and Pattern Recognition, celebrada en Salt Lake City (UT, USA), del 18 al 23 de junio de 2018, We propose a method for predicting the 3D shape of a deformable surface from a single view. By contrast with previous approaches, we do not need a pre-registered template of the surface, and our method is robust to the lack of texture and partial occlusions. At the core of our approach is a geometry-aware deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image. We train this architecture in an end-to-end manner using a large dataset of synthetic renderings of shapes under different levels of deformation, material properties, textures and lighting conditions. We evaluate our approach on a test split of this dataset and available real benchmarks, consistently improving state-of-the-art solutions with a significantly lower computational time., This work is supported in part by a Google Faculty Research Award, by the Spanish Ministry of Science and Innovation under projects HuMoUR TIN2017- 90086-R, ColRobTransp DPI2016-78957 and Mar´ıa de Maeztu Seal of Excellence MDM-2016-0656; and by the EU project AEROARMS ICT-2014-1-644271. We also thank Nvidia for hardware donation.

Doctoral Programme: Automàtica, Robòtica i Visió, [EN]: Although there has been remarkable progress in the pose estimation literature, there are still a number of limitations when existing algorithms must be applied in everyday applications, especially in uncontrolled environments. This thesis has addressed some of these limitations, computing the pose for uncalibrated cameras, computing the pose without knowing the correspondence between 20 and 30 points, computing the pose when the points of interest are unreliable and computing the pose using only depth data. The problems addressed, and consequently their contributions, have been analyzed in order of increasing complexity. At each new stage of the doctoral thesis existing restrictions for obtaining 30 camera pose increased. The thesis has consisted of four parts on which we will define the contributions made to the field of Computer Vision. The first contribution of the doctoral thesis has focused on providing a technique for obtaining the pose of an uncalibrated camera more robust and accurate than existing approaches. By the re-formulation of the equations used in calibrated perspectives methods and by studying numerical stability we obtained an extended equation formulation that offered a closed solution to the problem with increased stability in the presence of noise compared to the state of the art. The second contribution of the thesis has focused on the fact that most algorithms are based on having a set of 20-30 correspondences. This task usually involves the extraction and matching of points of interest. In this thesis it we have developed an algorithm that solves the estimation of correspondences between points and estimate the pose of the camera together, all this in an uncalibrated context. By solving both problems together you can optimize the steps we take much better than by just solving them separately. In articles published as a result of this work we have shown the advantages inherent in this approach. The third contribution of the thesis has been to provide a solution for estimating the pose of the camera in extreme situations where the image quality is very deteriorated. This is possible through the use of learning techniques from high-quality data and 30 models of the environment and the objects. This approach is based on the notion that by learning from high-quality data we can obtain detectors that are able to recognize objects in the worst circumstances because they know in depth what defines the object in question. The fourth contribution of the thesis is the creation of a pose estimation method that does not require color information, only depth. By defining local volumetric dense appearance and performing a dense feature extraction over the depth image. Once the dense feature sampling is obtained we perform an energy minimisation taking into account the pairwise terms between individual features. We obtain accuracy comparable to state of the art methods while performing atan arder of magnitude less time per image. The sum of the above contributions in 30 pose estimation have improved 30 reconstruction tools such as robotic vision and relocation in 30 maps. All contributions have been published in international journals and conferences of reputed scientific prestige in the area., [ES]: Aunque ha habido un progreso notable en la literatura de estimación de pose, todavía hay un número de limitaciones cuando los algoritmos existentes deben ser aplicados en aplicaciones de uso diario, especialmente en ambientes no controlados. En esta tesis se han abordado algunas de estas limitaciones, la computación de la pose para cámaras no calibradas, la computación de la pose sin conocer la correspondencia entre puntos 20 y 30, la computación de la pose cuando los puntos de interés no son fiables y la computación de la pose usando exclusivamente datos de profundidad. Los problemas abordados, y en consecuencia las contribuciones aportadas, han sido analizados en orden creciente de complejidad. En cada nueva etapa de la tesis doctoral se incrementaban las restricciones existentes para la obtención de la pose 30 de la cámara. La tesis ha constado de cuatro partes sobre las que pasaremos a definir las contribuciones realizadas al área de la Visión por Computador. La primera contribución de la tesis doctoral se ha centrado en ofrecer una técnica para la obtención de la pose de una cámara perspectiva sin calibrar más robusta y precisa que los existentes. Mediante la re-formulación de las ecuaciones perspectivas usadas en métodos calibrados y el estudio de la estabilidad numérica de las mismas se ha obtenido una formulación extendida de las ecuaciones perspectivas que ofrece una solución cerrada al problema y una mayor estabilidad en presencia de ruido. La segunda contribución de la tesis se ha centrado en el hecho de que la mayoría de los algoritmos se basan en tener un conjunto de correspondencias 20-30. Esta tarea implica generalmente la extracción y emparejamiento de puntos de interés. En esta tesis doctoral se ha desarrollado un algoritmo que aborda la estimación de las correspondencias entre puntos y estimación de la pose de la cámara de manera conjunta. Al resolver ambos problemas conjuntamente se puede optimizar los pasos a tomar mucho mejor que resolviéndolos por separado. En los trabajos publicados a raíz de este trabajo se han mostrado las ventajas inherentes a esta aproximación al problema. La tercera contribución de la tesis ha sido la de aportar una solución para la estimación de la pose de la cámara en situaciones extremas en las que la calidad de la imagen se encuentra muy deteriorada. Esto es posible mediante el uso de técnicas de aprendizaje a partir de datos de alta calidad y modelos 30 del entorno y los objetos presentes. Esta aproximación se basa en la noción de que a partir de un aprendizaje sobre datos de alta calidad se pueden obtener detectores que son capaces de reconocer los objetos en las peores circunstancias ya que conocen en profundidad aquello que define al objeto en cuestión. La cuarta contribución de la tesis es la creación de un método de estimación de pose que no requiere de información de color, solamente profundidad. Mediante una definición de apariencia volumétrica local y la extracción densa de características en la imagen de profundidad se obtiene un método comparable en precisión al estado de la cuestión pero un orden de magnitud mas rápido. La suma de las contribuciones anteriores en las tareas de estimación de pose 30 han posibilitado la mejora en las herramientas de reconstrucción 30, visión robótica y relocalización en mapas 30. Todas las contribuciones han sido publicadas en revistas y congresos internacionales y de reputado prestigio científico en el área.

Master’s Thesis: Master’s degree in Automatic Control and Robotics. Treball de Fi de Master has been carried out at: Escola Tècnica Superior d’Enginyeria Industrial de Barcelona of Universitat Politècnica de Catalunya., Recovering deformable 3D motion from temporal 2D point tracks in a monocular video is an open problem with many everyday applications throughout science and industry, or the new augmented reality. Recently, several techniques have been proposed to deal the problem called Non-Rigid Structure from Motion (NRSfM), however, they can exhibit poor reconstruction performance on complex motion. In this project, we will analyze these situations for primitive human actions such as walk, run, sit, jump, etc. on different scenarios, reviewing first the current techniques to finally present our novel method. This approach is able to model complex motion into a union of subspaces, rather than the summation occurring in standard low-rank shape methods, allowing better reconstruction accuracy. Experiments in a wide range of sequences and types of motion illustrate the benefits of this new approach.

Tesis doctoral presentada para lograr el título de Doctor por la Universidad Politécnica de Cataluña, programa de doctorado: Automatic Control, Robotics and Computer Vision.--2021-10-13, [EN] 3D reconstruction and image synthesis are two of the main pillars in computer vision. Early works focused on simple tasks such as multi-view reconstruction and texture synthesis. With the spur of Deep Learning, the field has rapidly progressed, making it possible to achieve more complex and high level tasks. For example, the 3D reconstruction results of traditional multi-view approaches are currently obtained with single view methods. Similarly, early pattern based texture synthesis works have resulted in techniques that allow generating novel high-resolution images. In this thesis we have developed a hierarchy of tools that cover all these range of problems, lying at the intersection of computer vision, graphics and machine learning. We tackle the problem of 3D reconstruction and synthesis in the wild. Importantly, we advocate for a paradigm in which not everything should be learned. Instead of applying Deep Learning naively we propose novel representations, layers and architectures that directly embed prior 3D geometric knowledge for the task of 3D reconstruction and synthesis. We apply these techniques to problems including scene/person reconstruction and photo-realistic rendering. We first address methods to reconstruct a scene and the clothed people in it while estimating the camera position. Then, we tackle image and video synthesis for clothed people in the wild. Finally, we bridge the gap between reconstruction and synthesis under the umbrella of a unique novel formulation. Extensive experiments conducted along this thesis show that the proposed techniques improve the performance of Deep Learning models in terms of the quality of the reconstructed 3D shapes / synthesised images, while reducing the amount of supervision and training data required to train them. In summary, we provide a variety of low, mid and high level algorithms that can be used to incorporate prior knowledge into different stages of the Deep Learning pipeline and improve performance in tasks of 3D reconstruction and image synthesis., [CAT] La reconstrucció 3D i la síntesi d'imatges són dos dels pilars fonamentals en visió per computador. Els estudis previs es centren en tasques senzilles com la reconstrucció amb informació multi-càmera i la síntesi de textures. Amb l'aparició del "Deep Learning", aquest camp ha progressat ràpidament, fent possible assolir tasques molt més complexes. Per exemple, per obtenir una reconstrucció 3D, tradicionalment s'utilitzaven mètodes multi-càmera, en canvi ara, es poden obtenir a partir d'una sola imatge. De la mateixa manera, els primers treballs de síntesi de textures basats en patrons han donat lloc a tècniques que permeten generar noves imatges completes en alta resolució. En aquesta tesi, hem desenvolupat una sèrie d'eines que cobreixen tot aquest ventall de problemes, situats en la intersecció entre la visió per computador, els gràfics i l'aprenentatge automàtic. Abordem el problema de la reconstrucció i la síntesi 3D en el món real. És important destacar que defensem un paradigma on no tot s'ha d'aprendre. Enlloc d'aplicar el "Deep Learning" de forma naïve, proposem representacions novedoses i arquitectures que incorporen directament els coneixements geomètrics ja existents per a aconseguir la reconstrucció 3D i la síntesi d'imatges. Nosaltres apliquem aquestes tècniques a problemes com ara la reconstrucció d'escenes/persones i a la renderització d'imatges fotorealistes. Primer abordem els mètodes per reconstruir una escena, les persones vestides que hi ha i la posició de la càmera. A continuació, abordem la síntesi d'imatges i vídeos de persones vestides en situacions quotidianes. I finalment, aconseguim, a través d'una nova formulació única, connectar la reconstrucció amb la síntesi. Els experiments realitzats al llarg d'aquesta tesi demostren que les tècniques proposades milloren el rendiment dels models de "Deepp Learning" pel que fa a la qualitat de les reconstruccions i les imatges sintetitzades alhora que redueixen la quantitat de dades necessàries per entrenar-los. En resum, proporcionem una varietat d'algoritmes de baix, mitjà i alt nivell que es poden utilitzar per incorporar els coneixements previs a les diferents etapes del "Deep Learning" i millorar el rendiment en tasques de reconstrucció 3D i síntesi d'imatges., This work has been partially supported by: - EU project AEROARMS - Aerial Robotics Inspection System, 2015-2019. Code: H2020-ICT-2014- 1-644271 - National Project ColRobTransp - Robotics Last Mile Transportation in Urban Areas, 2016-2019. Code: DPI2016-78957-R - Google Research Award - Geometry-aware CNNs for Non-Rigid Reconstruction, 2017-2018 - Amazon Research Award - Geometry-aware 3D Human Body Animation from Still Photos, 2018- 2019 - Google Research Award - GANimation3D: Unsupervised 3D Face Animation from Monocular Images, 2019-2020

Tesis doctoral presentada para lograr el título de Doctor por la Universidad Politécnica de Cataluña.--2021-05-17, This dissertation intends to provide theoretical and practical contributions on the development of deep-learning algorithms for autonomous driving applications. The research is motivated by the need of deep neural networks (DNNs) to get a full understanding of the surrounding area and to be executed on real driving scenarios with real vehicles equipped with specific hardware, such as memory constrained (DSP or GPU platforms) or multiple optical sensors, which constraints the algorithm’s development forcing the designed deep networks to be accurate, with minimum number of operations and low-memory consumption. The main objective of this thesis is, on one hand, the research in the actual limitations of Deep Learning based algorithms that prevent them of being integrated in nowadays’ ADAS (Advanced Driver- Assistance Systems) functionalities, and on the other hand, the design and implementation of Deep Learning algorithms able to overcome such constraints to be applied on real autonomous driving scenarios, enabling their integration in low-memory hardware platforms and avoiding sensor redundancy. Deep Learning applications have been widely exploited over the last years but have some weak points that need to be faced and overcame in order to fully integrate Deep Learning into the development process of big manufacturers or automotive companies, like the time needed to design, train and validate and optimal network for a specific application or the vast knowledge from the required experts to tune hyperparameters of predefined networks in order to make them executable in the target platform and to obtain the biggest advantage of the hardware resources. During this thesis, we have addressed these topics and focused on the implementations of breakthroughs that would help in the industrial integration of Deep Learning based applications in the automobile industry. To this end, in the first part of this Thesis we focus on new approaches for vehicle pose estimation using only RGB planar images, avoiding the usage of other sensors that would lead to more costly and hard-to-manage systems. Furthermore, we investigate on the actual limitation of other hardware platforms different that the widely used GPUs (Graphical Processing Unit), such as FPGAs (Field-programmable Gate Array), to host a self-implemented algorithm that understands the surrounding scene of the ego-vehicle specifically designed to run fast and accurate, optimizing the FPGA resources. Finally, once we have done a deep research about the current constraints in the integration of Deep Learning based algorithms in the industry, we propose a new method to overcome them. We go deeper in the design of light DNN (Deep Neural Network) architectures and propose a framework for designing automatically DNNs to run on hardware constrained platforms. We test our development on commercial System on Chips (SoCs) such as Texas Instruments TDA2x SoC family, which are very typical in the automobile industry, obtaining remarkable results in image classification tasks in terms of accuracy, inference time and search time.

Tesis llevada a cabo para conseguir el grado de Doctor por la Universidad Politécnica de Cataluña.--2020-03-09, [EN] This thesis addresses the problem of improving text spotting systems, which aim to detect and recognize text in unrestricted images (e.g. a street sign, an advertisement, a bus destination, etc.). The goal is to improve the performance of off-the-shelf vision systems by exploiting the semantic information derived from the image itself. The rationale is that knowing the content of the image or the visual context can help to decide which words are the correct andidate words. For example, the fact that an image shows a coffee shop makes it more likely that a word on a signboard reads as Dunkin and not unkind. We address this problem by drawing on successful developments in natural language processing and machine learning, in particular, learning to re-rank and neural networks, to present post-process frameworks that improve state-of-the-art text spotting systems without the need for costly data-driven re-training or tuning procedures. Discovering the degree of semantic relatedness of candidate words and their image context is a task related to assessing the semantic similarity between words or text fragments. However, semantic relatedness is more general than similarity (e.g. car, road, and traffic light are related but not similar) and requires certain adaptations. To meet the requirements of these broader perspectives of semantic similarity, we develop two approaches to learn the semantic related-ness of the spotted word and its environmental context: word-to-word (object) or word-to-sentence (caption). In the word-to-word approach, word embed-ding based re-rankers are developed. The re-ranker takes the words from the text spotting baseline and re-ranks them based on the visual context from the object classifier. For the second, an end-to-end neural approach is designed to drive image description (caption) at the sentence-level as well as the word-level (objects) and re-rank them based not only on the visual context but also on the co-occurrence between them. As an additional contribution, to meet the requirements of data-driven ap-proaches such as neural networks, we propose a visual context dataset for this task, in which the publicly available COCO-text dataset [Veit et al. 2016] has been extended with information about the scene (including the objects and places appearing in the image) to enable researchers to include the semantic relations between texts and scene in their Text Spotting systems, and to offer a common evaluation baseline for such approaches., [CA] Aquesta tesi aborda el problema de millorar els sistemes de reconeixement de text, que permeten detectar i reconèixer text en imatges no restringides (per exemple, un cartell al carrer, un anunci, una destinació d’autobús, etc.). L’objectiu és millorar el rendiment dels sistemes de visió existents explotant la informació semàntica derivada de la pròpia imatge. La idea principal és que conèixer el contingut de la imatge o el context visual en el que un text apareix, pot ajudar a decidir quines són les paraules correctes. Per exemple, el fet que una imatge mostri una cafeteria fa que sigui més probable que una paraula en un rètol es llegeixi com a Dunkin que no pas com unkind. Abordem aquest problema recorrent a avenços en el processament del llenguatge natural i l’aprenentatge automàtic, en particular, aprenent re-rankers i xarxes neuronals, per presentar solucions de postprocés que milloren els sistemes de l’estat de l’art de reconeixement de text, sense necessitat de costosos procediments de reentrenament o afinació que requereixin grans quantitats de dades. Descobrir el grau de relació semàntica entre les paraules candidates i el seu context d’imatge és una tasca relacionada amb l’avaluació de la semblança semàntica entre paraules o fragments de text. Tanmateix, determinar l’existència d’una relació semàntica és una tasca més general que avaluar la semblança (per exemple, cotxe, carretera i semàfor estan relacionats però no són similars) i per tant els mètodes existents requereixen certes adaptacions. Per satisfer els requisits d’aquestes perspectives més àmplies de relació semàntica, desenvolupem dos enfocaments per aprendre la relació semàntica de la paraula reconeguda i el seu context: paraula-a-paraula (amb els objectes a la imatge) o paraula-a-frase (subtítol de la imatge). En l’enfocament de paraula-a-paraula s’usen re-rankers basats en word-embeddings. El re-ranker pren les paraules proposades pel sistema base i les torna a reordenar en funció del context visual proporcionat pel classificador d’objectes. Per al segon cas, s’ha dissenyat un enfocament neuronal d’extrem a extrem per explotar la descripció de la imatge (subtítol) tant a nivell de frase com a nivell de paraula i re-ordenar les paraules candidates basant-se tant en el context visual com en les co-ocurrències amb el subtítol. Com a contribució addicional, per satisfer els requisits dels enfocs basats en dades com ara les xarxes neuronals, presentem un conjunt de dades de contextos visuals per a aquesta tasca, en el què el conjunt de dades COCO-text disponible públicament [Veit et al. 2016] s’ha ampliat amb informació sobre l’escena (inclosos els objectes i els llocs que apareixen a la imatge) per permetre als investigadors incloure les relacions semàntiques entre textos i escena als seus sistemes de reconeixement de text, i oferir una base d’avaluació comuna per a aquests enfocaments

The objective of this work is to obtain an end-to-end solution which predicts human motion and shape from a given video by extracting its pose in the Wild. Given incomplete human motion sequences our goal is to predict and visualize the following frames of those sequences, with realistic human shape

The popularity of wearable cameras is steadily increasing, both for entertainment and productivity purposes. Understanding the footage and inferring the body pose of the camera wearer can be of great importance in many fields, like medicine or robotics. It could help in monitoring rehabilitating patients at home from the hospital, in determining the acts of a law enforcement agent from the body camera or, in robotics, simplifying imitation learning based on video input or robot to worker coordination by estimating the posture of the operator. In this project, we aim to build a human motion dataset acquired indoors and outdoors using a GoPro and MVN Awinda, a movement tracking system based on inertial sensors that provide the 3D human pose. Next, the dataset has been used to train a Deep Neural Network to classify a sequence of frames in the task that it’s being performed (walking, running, ...). Finally, a model for estimating the 3D pose of the camera wearer at each frame is proposed based on the same structure than the first network. The dataset is made of 300,000 frames, captured from seven different people, each one perform-ing 5-6 tasks in different scenarios, both indoors and outdoors. Each sequence of video frames has a synchronized sequence of 3D poses associated to it. Every 3D pose is composed of 23 segments. The vast majority of the code created and used in this project can be found in the GitHub repository for the project: https://github.com/BielColl/Human-Motion-Dataset-in-the-Wild-MAT-. Due to its size, the dataset built is not posted.

The present Master Thesis describes a new Pose Estimation method based on Convolutional Neural Networks (CNN). This method divides the three-dimensional space in several regions and, given an input image, returns the region where the camera is located. The first step is to create synthetic images of the object simulating a camera located at di↵erent points around it. The CNN is pre-trained with these thousands of synthetic images of the object model. Then, we compute the pose of the object in hundreds of real images, and apply transfer learning with these labeled real images over the existing CNN, in order to refine the weights of the neurons and improve the network behaviour against real input images. Along with this deep learning approach, other techniques have been used trying to improve the quality of the results, such as the classical sliding window or a more recent class-generic object detector called objectness. It is tested with a 2D-model in order to ease the labeling process of the real images. This document outlines all the steps followed to create and test the method, and finally compares it against a state-of-the-art method at di↵erent scales and levels of blurring.

El presente proyecto tiene como objetivo principal la implementación de una aplicación para dispositivos Android. Esta aplicación, permite calcular en tiempo real, la pose de la cámara respecto a un objeto. La posición y la orientación (rotación y traslación) es lo que se entiende por pose. A lo largo del documento se explican las técnicas y algoritmos usados para calcular la pose en 3D de la cámara. Para calcular la pose de la cámara en tiempo real, se tiene que entender cómo se proyecta un objeto físico en una imagen. Por esta razón, se estudiarán los conceptos de la proyección en perspectiva y el modelo pinhole de una cámara, que define la relación entre las coordenadas 3D y su correspondiente proyección 2D. La idea general del algoritmo, consiste en encontrar la relación entre las coordenadas del objeto en el mundo real, con las coordenadas del objeto en el plano de la imagen. Esta relación geométrica, llamada matriz de proyección, es muy importante en el mundo de la visión por computador y tiene en cuenta los parámetros internos y externos de la cámara (modelo pinhole). La aplicación desarrollada, se podría utilizar para realizar importantes aplicaciones de realidad aumentada. Algunos algoritmos que se utilizan para encontrar la relación comentada son: el algoritmo de Möller-Trumbore, Ransac, EPnP y UPnP. Estos algoritmos se explicarán detenidamente durante el proyecto, así como otros. Para llevar a cabo el proyecto, se han utilizado dos lenguajes de programación, Java y C++, con el propósito de combinar las ventajas de cada uno. Se han implementado en la plataforma Linux utilizando el entorno de desarrollo llamado Eclipse. La implementación se ha logrado con ayuda de las bibliotecas de OpenCV. Bibliotecas de uso libre de visión por computador que permiten crear aplicaciones en el mundo de la visión digital.

Universitat Politècnica de Catalunya. Escola Tècnica Superior d’Enginyeria Industrial de Barcelona (ETSEIB). Systems Engineering, Automation and Industrial Informatics (ESAII). Master’s Degree in Automatic Control and Robotics. Insitut de Robòtica i Informàtica Industrial (CSIC-UPC)., In this project, I present the implementation of a 3D pose estimation algorithm for rigid objects considering a single monocular camera. The algorithm based on the matching between natural feature points and a textured 3D model, recovers in an efficient way the 3D pose of a given object using a PnP method. Furthermore, during this project a C++ implementation of the UPnP approach published by the supervisors of this project has been done. Both the algorithm and the UPnP source codes have been included in the OpenCV library. The first as a tutorial explaining how developers could implement this kind of algorithms, and the second as an extension of the Camera Calibration and 3D Reconstruction module. In order to ensure the quality of the code, both passed the accuracy and performance tests imposed by the organization before merging the code in the new released version.

Trabajo presentado en la IEEE/CVF Conference on Computer Vision and Pattern Recognition, celebrada en Salt Lake City (UT, USA), del 18 al 23 de junio de 2018, We present a novel approach for synthesizing photorealistic images of people in arbitrary poses using generative adversarial learning. Given an input image of a person and a desired pose represented by a 2D skeleton, our model renders the image of the same person under the new pose, synthesizing novel views of the parts visible in the input image and hallucinating those that are not seen. This problem has recently been addressed in a supervised manner [16, 35], i.e., during training the ground truth images under the new poses are given to the network. We go beyond these approaches by proposing a fully unsupervised strategy. We tackle this challenging scenario by splitting the problem into two principal subtasks. First, we consider a pose conditioned bidirectional generator that maps back the initially rendered image to the original pose, hence being directly comparable to the input image without the need to resort to any training image. Second, we devise a novel loss function that incorporates content and style terms, and aims at producing images of high perceptual quality. Extensive experiments conducted on the DeepFashion dataset demonstrate that the images rendered by our model are very close in appearance to those obtained by fully supervised approaches., This work is supported in part by a Google Faculty Research Award, by the Spanish Ministry of Science and Innovation under projects HuMoUR TIN2017- 90086-R, ColRobTransp DPI2016-78957 and Mar´ıa de Maeztu Seal of Excellence MDM-2016-0656; and by the EU project AEROARMS ICT-2014-1-644271. We also thank Nvidia for hardware donation under the GPU Grant Program.