Search

First proposed over 20 years ago, 3D morphable models (3DMMs) provide an insight into the statistical shape variations of a given shape class. Craniofacial anomalies represent a diverse class of conditions in which one or more of the sutures in the infant cranium fuses prematurely, causing a deviation in the growth pattern of the skull from that of the unaffected population. 3DMMs present an opportunity to analyse these shape deviations from a statistical perspective and can be used for the development of diagnostic and surgical planning tools. For this to be achieved, however, several boundary limitations must be overcome. Constructing statistical shape models for a population affected by craniosynostosis is by no means a straightforward task. As is suggested by the description "craniofacial anomalies", instances of craniosynostosis are rare. Consequently, it can be difficult to collect sufficient quantities of data to accurately model the shape variations of the affected population. While 3D facial and cranial datasets are becoming increasingly available, these datasets tend to be comprised of adolescent and adult samples and few contain instances of infants and young children. Even when adequate 3D data has been collected, annotation is expensive and time consuming. Finally, while mesh autoencoders have proven effective for accurate mesh reconstruction and novel shape generation, there are many advances yet to be made in this domain. This thesis aims to make progress addressing the aforementioned issues in a number of ways. To address some of the existing limitations surrounding sparse 3D annotation, we present two methods for landmark localisation in 3D point clouds: i) point cloud segmentation and offset vector prediction are combined to localise landmarks within the input cloud and ii) the concept of convolutional pose machines is extended to predict landmark heatmaps for 3D facial point clouds. Both methods are shown to be robust to a range of input point cloud sizes and can be readily applied for landmark localisation in any shape class given appropriate training data. To aid in surgical reconstruction procedures for ear microtia we propose an approach to infer the underlying ear cartilage shape from that of the external ear soft tissue. We achieve this by generalising the convolutional mesh autoencoder framework such that the shape class and mesh topology of the encoder and decoder need not be identical. We further introduce an intersection loss to enforce the spatial relationship between the outer ear and underlying cartilage and improve the robustness of predicted results. As normal models of the paediatric population are limited, we collect a dataset of CT scans for infants aged between 0 and 48 months. From this dataset we construct 3DMMs of the face, head, skull, and mandible. By collating data from multiple sources, we build facial and head models to aid in the diagnosis of craniofacial anomalies related to mutations in the FGFR gene. High diagnostic accuracy is achieved, and experiments demonstrate that the facial features contain important information for the correct diagnosis of syndromic craniosynostosis. Finally, neural approaches to the construction of 3DMMs can result in a loss of interpretability and regularisation in the model latent space. To address this, we propose PCA retargeting, a method for expressing linear PCA models as convolutional mesh autoencoders and thereby retaining the gaussianity of the latent space.

Graphs serve as a versatile mathematical abstraction of real-world phenomena in numerous scientific disciplines. This thesis is part of the Geometric Deep Learning subject area, a family of learning paradigms, that capitalise on the increasing volume of non-Euclidean data so as to solve real-world tasks in a data-driven manner. In particular, we focus on the topic of graph function approximation using neural networks, which lies at the heart of many relevant methods. In the first part of the thesis, we contribute to the understanding and design of Graph Neural Networks (GNNs). Initially, we investigate the problem of learning on signals supported on a fixed graph. We show that treating graph signals as general graph spaces is restrictive and conventional GNNs have limited expressivity. Instead, we expose a more enlightening perspective by drawing parallels between graph signals and signals on Euclidean grids, such as images and audio. Accordingly, we propose a permutation-sensitive GNN based on an operator analogous to shifts in grids and instantiate it on 3D meshes for shape modelling (Spiral Convolutions). Following, we focus on learning on general graph spaces and in particular on functions that are invariant to graph isomorphism. We identify a fundamental trade-off between invariance, expressivity and computational complexity, which we address with a symmetry-breaking mechanism based on substructure encodings (Graph Substructure Networks). Substructures are shown to be a powerful tool that provably improves expressivity while controlling computational complexity, and a useful inductive bias in network science and chemistry. In the second part of the thesis, we discuss the problem of graph compression, where we analyse the information-theoretic principles and the connections with graph generative models. We show that another inevitable trade-off surfaces, now between computational complexity and compression quality, due to graph isomorphism. We propose a substructure-based dictionary coder - Partition and Code (PnC) - with theoretical guarantees that can be adapted to different graph distributions by estimating its parameters from observations. Additionally, contrary to the majority of neural compressors, PnC is parameter and sample efficient and is therefore of wide practical relevance. Finally, within this framework, substructures are further illustrated as a decisive archetype for learning problems on graph spaces.

In this thesis, we explore the problem of facial attribute analysis and editing in images, from the scope of two major fields of Machine Learning, those of Components Analysis (CA) and Deep Learning (DL). First, we present a CA method for analysing and editing facial data. Then, we present a DL algorithm for animating facial images according to expressions and speech. Finally, we present a method for improving gaze estimation generalisation to unseen image domains and showcase applications to eye gaze editing. Although CA methods are able to capture only linear relationships in data, they can still be useful with well-aligned data, such as UV maps of facial texture. In this Thesis, we propose robust extensions of the Joint and Individual Variance Explained (JIVE) method, for the recovery of joint and individual components in visual facial data, captured in unconstrained conditions and possibly containing sparse non-Gaussian errors and missing data. We demonstrate the effectiveness of the proposed methods to several computer vision applications, namely facial expression synthesis and 2D and 3D face age progression in-the-wild. CA methods usually fall short in image generation, as they fail to generate details. On the contrary, Image-to-image (i2i) translation, which is the problem of translating images between image domains, has recently seen remarkable progress since the advent of DL and Generative Adversarial Networks (GANs). In this Thesis, we study the problem of i2i translation, under a set of continuous parameters that correspond to statistical blendshape models of facial motion. We show that it is possible to edit facial images according to expression and speech blendshapes using ''sliders'', which are more flexible than discrete expressions or action units. Lastly, realistically animating gaze is crucial for achieving high quality facial animations. To this end, large datasets of faces with gaze annotations are required for training. In this Thesis, we present a weakly-supervised method for improving gaze estimation generalization to unseen domains, by harnessing arbitrary unlabelled ''in-the-wild'' face images. Unlike previous methods, we tackle gaze estimation as end-to-end, dense 3D reconstruction of eyes and experimentally validate the benefits of this choice. Particularly, we show improvements in semi-supervised and cross-dataset gaze estimation. Finally, we showcase how our methods can be employed for training efficient models for gaze editing.

The advent of Deep Learning has led to numerous breakthroughs in the field of Computer Vision. Over the last decade, a significant amount of research has been undertaken towards designing neural networks for visual data analysis. At the same time, rapid advancements have been made towards the direction of deep generative modeling, especially after the introduction of Generative Adversarial Networks (GANs), which have shown particularly promising results when it comes to synthesising visual data. Since then, considerable attention has been devoted to the problem of photo-realistic human face animation due to its wide range of applications, including image and video editing, virtual assistance, social media, teleconferencing, and augmented reality. The objective of this thesis is to make progress towards generating photo-realistic videos of human faces. To that end, we propose novel generative algorithms that provide explicit control over the facial expression and head pose of synthesised subjects. Despite the major advances in face reenactment and motion transfer, current methods struggle to generate video portraits that are indistinguishable from real data. In this work, we aim to overcome the limitations of existing approaches, by combining concepts from deep generative networks and video-to-video translation with 3D face modelling, and more specifically by capitalising on prior knowledge of faces that is enclosed within statistical models such as 3D Morphable Models (3DMMs). In the first part of this thesis, we introduce a person-specific system that performs full head reenactment using ideas from video-to-video translation. Subsequently, we propose a novel approach to controllable video portrait synthesis, inspired from Implicit Neural Representations (INR). In the second part of the thesis, we focus on person-agnostic methods and present a GAN-based framework that performs video portrait reconstruction, full head reenactment, expression editing, novel pose synthesis and face frontalisation.

Deformable models are powerful tools for modelling the 3D shape variations for a class of objects. However, currently the application and performance of deformable models for human body are restricted due to the limitations in current 3D datasets, annotations, and the model formulation itself. In this thesis, we address the issue by making the following contributions in the field of 3D human body modelling, monocular reconstruction and data collection/annotation. Firstly, we propose a deep mesh convolutional network based deformable model for 3D human body. We demonstrate the merit of this model in the task of monocular human mesh recovery. While outperforming current state of the art models in mesh recovery accuracy, the model is also light weighted and more flexible as it can be trained end-to-end and fine-tuned for a specific task. A second contribution is a bone level skinned model of 3D human mesh, in which bone modelling and identity-specific variation modelling are decoupled. Such formulation allows the use of mesh convolutional networks for capturing detailed identity specific variations, while explicitly controlling and modelling the pose variations through linear blend skinning with built-in motion constraints. This formulation not only significantly increases the accuracy in 3D human mesh reconstruction, but also facilitates accurate in the wild character animation and retargetting. Finally we present a large scale dataset of over 1.3 million 3D human body scans in daily clothing. The dataset contains over 12 hours of 4D recordings at 30 FPS, consisting of 7566 dynamic sequences of 3D meshes from 4205 subjects. We propose a fast and accurate sequence registration pipeline which facilitates markerless motion capture and automatic dense annotation for the raw scans, leading to automatic synthetic image and annotation generation that boosts the performance for tasks such as monocular human mesh reconstruction.

It has been more than 20 year since the introduction of 3D morphable models (3DMM) in the computer vision literature. They were proposed as a face representation based on principal components analysis for the task of image analysis, photorealist-manipulation, and 3D reconstruction from single images. Even so, to this date, the applications of such models are limited by a number of factors. Firstly, training correctly 3DMMs require a vast amount of 3D data that most of the times are not publicly available to the research community due to increasingly stringent data protection regulations. Hence, it is extremely difficult to combine and enrich multiple attributes of the human face/head without the initial 3D images. Additionally, many 3DMMs utilize different templates that describe distinct parts of the human face/head (\ie~face, cranium, ears, eyes) that partly overlap with each other and capture statistical variations which are extremely difficult to incorporate into one single universal morphable model. Moreover, despite the increasing level of detail in the 3D face reconstruction from in-the-wild images, mainly attributed to recent advancements in deep learning, non of the available methods in the literature deal with the human tongue which is important for speech dynamics and improves the realness of the oral cavity. Finally, there is limited work on 3D facial geometric enchantments and translations from different capturing systems due to extremely limited availability of 3D dasasets tailored for this task. This thesis aims at tackling these shortcomings in all four domains. A novel approach on how to combine and enrich existing 3DMMs without the underline raw data is proposed. We introduce two methods for solving this problem: i. use a regressor to complete missing parts of one model using the other, ii. use a Gaussian Process framework to blend covariance matrices from multiple models. We show case our approach by combining existing face and head 3DMMs with different templates and statistical variations. Furthermore, we introduce to the research community the first Universal Head Model (UHM) which holds important statistical variation across all key structures of the human head that have an important contribution to to the appearance and identity of a person. We later show case how this model is used to create full head appearances from single in-the-wild images, thus making significant improvements toward the step of realist human head digitization from data-deficient sources. Additionally, we present the first method that accurately reconstructs the human tongue from single images by utilizing a novel generative framework which models directly the highly deformable surface of the human tongue and seamlessly merges it with our universal head model for more realist representations of the oral cavity dynamics. Lastly, in this thesis, it is presented a novel generative pipeline capable of converting and enhancing low to high quality 3D facial scans. This will potentially aid depth sensor applications by increasing the quality of the output data while maintaining a low cost. It is also shown that the proposed framework can be extended to handle translations between various expressions on demand.

Face recognition is the most prominent bio-metric technique for identity authentication and is widely used in applications, such as access control, finance, law enforcement, and public security. Even though face biometrics is widely deployed, the current face recognition system is still limited to applications where strict control is imposed on the process of face image capture. This thesis targets on develop unconstrained face recognition technology, which is robust to a range of appearance variations and degradation factors, based on recent advances in deep neural networks and sophisticated prior information conveyed by 3D face models. We develop a complete pipeline for deep face recognition in real-life, naturalistic conditions (in-the-wild), covering the main steps (e.g. face detection, face alignment and face feature embedding). First, we innovatively propose RetinaFace, which unifies face box prediction, 2D facial landmark localisation and 3D vertices regression under one common target: point regression on the image plane. The proposed 3D mesh regression branch can be easily incorporated, without any optimisation difficulty, in parallel with the existing box and 2D landmark regression branches during joint training. RetinaFace can achieve state-of-the-art face detection while being efficient through single-shot inference. Then, we propose stacked dense U-Nets for face alignment under large pose variations. Our method employs a novel scale aggregation network topology structure and a channel aggregation building block to improve the model's capacity without sacrificing the computational complexity and model size. Extensive experiments on many in-the-wild datasets, validate the robustness of the proposed full-pose facial landmark localisation method. Through sampling the image using the fitted 3D face model, a facial UV can be created. Unfortunately, due to self-occlusion, such a UV map is always incomplete. We devise a meticulously designed architecture that combines local and global adversarial DCNNs to learn an identity-preserving facial UV completion model. We demonstrate that by attaching the completed UV to the fitted mesh and generating instances of arbitrary poses, we can increase pose variations for training deep face recognition models, and minimise pose discrepancy during testing, which lead to pose-invariant face recognition. For large-scale face recognition, we propose an Additive Angular Margin Loss (ArcFace) to obtain highly discriminative features. The proposed ArcFace has a clear geometric interpretation due to its exact correspondence to geodesic distance on a hypersphere. Besides, ArcFace only adds negligible computational complexity during training and current servers can easily support millions of identities. ArcFace achieves state-of-the-art performance on many face recognition benchmarks including large-scale image and video datasets. Based on the proposed efficient face localisation method (RetinaFace) and effective face feature embedding method (ArcFace), we achieve state-of-the-art performance in the global Face Recognition Vendor Test (FRVT). We have also released our solution to facilitate future research.

This Thesis revolves around datasets and algorithms, with a focus on generative modelling. In particular, we first turn our attention to a novel, multi-attribute, 2D facial dataset. We then present deterministic as well as probabilistic Component Analysis (CA) techniques which can be applied to multi-attribute 2D as well as 3D data. We finally present deep learning generative approaches specially designed to manipulate 3D facial data. Most 2D facial datasets that are available in the literature, are: a) automatically or semi-automatically collected and thus contain noisy labels, hindering the benchmarking and comparisons between algorithms. Moreover, they are not annotated for multiple attributes. In the first part of the Thesis, we present the first manually collected and annotated database, which contains labels for multiple attributes. As we demonstrate in a series of experiments, it can be used in a number of applications ranging from image translation to age-invariant face recognition. Moving on, we turn our attention to CA methodologies. CA approaches, although being able to only capture linear relationships between data, can still be proven to be efficient in data such as UV maps or 3D data registered in a common template, since they are well aligned. The introduction of more complex datasets in the literature, which contain labels for multiple attributes, naturally brought the need for novel algorithms that can simultaneously handle multiple attributes. In this Thesis, we cover novel CA approaches which are specifically designed to be utilised in datasets annotated with respect to multiple attributes and can be used in a variety of tasks, such as 2D image denoising and translation, as well as 3D data generation and identification. Nevertheless, while CA methods are indeed efficient when handling registered 3D facial data, linear 3D generative models lack details when it comes to reconstructing or generating finer facial characteristics. To alleviate this, in the final part of this Thesis we propose a novel generative framework harnessing the power of Generative Adversarial Networks.

Affect recognition based on a subject's facial expressions has been a topic of major research in the attempt to generate machines that can understand the way subjects feel, act and react. In the past, due to the unavailability of large amounts of data captured in real-life situations, research has mainly focused on controlled environments. However, recently, social media and platforms have been widely used. Moreover, deep learning has emerged as a means to solve visual analysis and recognition problems. This Ph.D. Thesis exploits these advances and makes significant contributions for affect analysis and recognition in-the-wild. We tackle affect analysis and recognition as a dual knowledge generation problem: i) we create new, large and rich in-the-wild databases and ii) we design and train novel deep neural architectures that are able to analyse affect over these databases and to successfully generalise their performance on other datasets. At first, we present the creation of Aff-Wild database annotated according to valence-arousal and an end-to-end CNN-RNN architecture, AffWildNet. Then we use AffWildNet as a robust prior for dimensional and categorical affect recognition and extend it by extracting low-/mid-/high-level latent information and analysing this via multiple RNNs. Additionally, we propose a novel loss function for DNN-based categorical affect recognition. Next, we generate Aff-Wild2, the first database containing annotations for all main behavior tasks: estimate Valence-Arousal; classify into Basic Expressions; detect Action Units. We develop multi-task and multi-modal extensions of AffWildNet by fusing these tasks and propose a novel holistic approach that utilises all existing databases with non-overlapping annotations and couples them through co-annotation and distribution matching. Finally, we present an approach for valence-arousal, or basic expressions' facial affect synthesis. We generate an image with a given affect, or a sequence of images with evolving affect, by annotating a 4-D database and utilising a 3-D morphable model.

The past few decades have witnessed substantial progress towards 3D facial modelling and reconstruction as it is high importance for many computer vision and graphics applications including Augmented/Virtual Reality (AR/VR), computer games, movie post-production, image/video editing, medical applications, etc. In the traditional approaches, facial texture and shape are represented as triangle mesh that can cover identity and expression variation with non-rigid deformation. A dataset of 3D face scans is then densely registered into a common topology in order to construct a linear statistical model. Such models are called 3D Morphable Models (3DMMs) and can be used for 3D face synthesization or reconstruction by a single or few 2D face images. The works presented in this thesis focus on the modernization of these traditional techniques in the light of recent advances of deep learning and thanks to the availability of large-scale datasets. Ever since the introduction of 3DMMs by over two decades, there has been a lot of progress on it and they are still considered as one of the best methodologies to model 3D faces. Nevertheless, there are still several aspects of it that need to be upgraded to the "deep era". Firstly, the conventional 3DMMs are built by linear statistical approaches such as Principal Component Analysis (PCA) which omits high-frequency information by its nature. While this does not curtail shape, which is often smooth in the original data, texture models are heavily afflicted by losing high-frequency details and photorealism. Secondly, the existing 3DMM fitting approaches rely on very primitive (i.e. RGB values, sparse landmarks) or hand-crafted features (i.e. HOG, SIFT) as supervision that are sensitive to "in-the-wild" images (i.e. lighting, pose, occlusion), or somewhat missing identity/expression resemblance with the target image. Finally, shape, texture, and expression modalities are separately modelled by ignoring the correlation among them, placing a fundamental limit to the synthesization of semantically meaningful 3D faces. Moreover, photorealistic 3D face synthesis has not been studied thoroughly in the literature. This thesis attempts to address the above-mentioned issues by harnessing the power of deep neural network and generative adversarial networks as explained below: Due to the linear texture models, many of the state-of-the-art methods are still not capable of reconstructing facial textures with high-frequency details. For this, we take a radically different approach and build a high-quality texture model by Generative Adversarial Networks (GANs) that preserves details. That is, we utilize GANs to train a very powerful generator of facial texture in the UV space. And then show that it is possible to employ this generator network as a statistical texture prior to 3DMM fitting. The resulting texture reconstructions are plausible and photorealistic as GANs are faithful to the real-data distribution in both low- and high- frequency domains. Then, we revisit the conventional 3DMM fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective. We propose to optimize the parameters with the supervision of pretrained deep identity features through our end-to-end differentiable framework. In order to be robust towards initialization and expedite the fitting process, we also propose a novel self-supervised regression-based approach. We demonstrate excellent 3D face reconstructions that are photorealistic and identity preserving and achieve for the first time, to the best of our knowledge, facial texture reconstruction with high-frequency details. In order to extend the non-linear texture model for photo-realistic 3D face synthesis, we present a methodology that generates high-quality texture, shape, and normals jointly. To do so, we propose a novel GAN that can generate data from different modalities while exploiting their correlations. Furthermore, we demonstrate how we can condition the generation on the expression and create faces with various facial expressions. Additionally, we study another approach for photo-realistic face synthesis by 3D guidance. This study proposes to generate 3D faces by linear 3DMM and then augment their 2D rendering by an image-to-image translation network to the photorealistic face domain. Both works demonstrate excellent photorealistic face synthesis and show that the generated faces are improving face recognition benchmarks as synthetic training data. Finally, we study expression reconstruction for personalized 3D face models where we improve generalization and robustness of expression encoding. First, we propose a 3D augmentation approach on 2D head-mounted camera images to increase robustness to perspective changes. And, we also propose to train generic expression encoder network by populating the number of identities with a novel multi-id personalized model training architecture in a self-supervised manner. Both approaches show promising results in both qualitative and quantitative experiments.

Dimensionality reduction and noise removal are fundamental machine learning tasks that are vital to artificial intelligence applications. Principal component analysis has long been utilised in computer vision to achieve the above mentioned goals. Recently, it has been enhanced in terms of robustness to outliers in robust principal component analysis. Both convex and non-convex programs have been developed to solve this new formulation, some with exact convergence guarantees. Its effectiveness can be witnessed in image and video applications ranging from image denoising and alignment to background separation and face recognition. However, robust principal component analysis is by no means perfect. This dissertation identifies its limitations, explores various promising options for improvement and validates the proposed algorithms on both synthetic and real-world datasets. Common algorithms approximate the NP-hard formulation of robust principal component analysis with convex envelopes. Though under certain assumptions exact recovery can be guaranteed, the relaxation margin is too big to be squandered. In this work, we propose to apply gradient descent on the Burer-Monteiro bilinear matrix factorisation to squeeze this margin given available subspaces. This non-convex approach improves upon conventional convex approaches both in terms of accuracy and speed. On the other hand, oftentimes there is accompanying side information when an observation is made. The ability to assimilate such auxiliary sources of data can ameliorate the recovery process. In this work, we investigate in-depth such possibilities for incorporating side information in restoring the true underlining low-rank component from gross sparse noise. Lastly, tensors, also known as multi-dimensional arrays, represent real-world data more naturally than matrices. It is thus advantageous to adapt robust principal component analysis to tensors. Since there is no exact equivalence between tensor rank and matrix rank, we employ the notions of Tucker rank and CP rank as our optimisation objectives. Overall, this dissertation carefully defines the problems when facing real-world computer vision challenges, extensively and impartially evaluates the state-of-the-art approaches, proposes novel solutions and provides sufficient validations on both simulated data and popular real-world datasets for various mainstream computer vision tasks.

Several factors contribute to the appearance of an object in a visual scene, including pose, illumination, and deformation, among others. Each factor accounts for a source of variability in the data. It is assumed that the multiplicative interactions of these factors emulate the entangled variability, giving rise to the rich structure of visual object appearance. Disentangling such unobserved factors from visual data is a challenging task, especially when the data have been captured in uncontrolled recording conditions (also referred to as "in-the-wild") and label information is not available. The work presented in this thesis focuses on disentangling the variations contained in visual data, in particular applied to 2D and 3D faces. The motivation behind this work lies in recent developments in the field, such as (i) the creation of large, visual databases for face analysis, with (ii) the need of extracting information without the use of labels and (iii) the need to deploy systems under demanding, real-world conditions. In the first part of this thesis, we present a method to synthesise plausible 3D expressions that preserve the identity of a target subject. This method is supervised as the model uses labels, in this case 3D facial meshes of people performing a defined set of facial expressions, to learn. The ability to synthesise an entire facial rig from a single neutral expression has a large range of applications both in computer graphics and computer vision, ranging from the ecient and cost-e↵ective creation of CG characters to scalable data generation for machine learning purposes. Unlike previous methods based on multilinear models, the proposed approach is capable to extrapolate well outside the sample pool, which allows it to accurately reproduce the identity of the target subject and create artefact-free expression shapes while requiring only a small input dataset. We introduce global-local multilinear models that leverage the strengths of expression-specific and identity-specific local models combined with coarse motion estimations from a global model. The expression-specific and identity-specific local models are built from di↵erent slices of the patch-wise local multilinear model. Experimental results show that we achieve high-quality, identity-preserving facial expression synthesis results that outperform existing methods both quantitatively and qualitatively. In the second part of this thesis, we investigate how the modes of variations from visual data can be extracted. Our assumption is that visual data has an underlying structure consisting of factors of variation and their interactions. Finding this structure and the factors is important as it would not only help us to better understand visual data but once obtained we can edit the factors for use in various applications. Shape from Shading and expression transfer are just two of the potential applications. To extract the factors of variation, several supervised methods have been proposed but they require both labels regarding the modes of variations and the same number of samples under all modes of variations. Therefore, their applicability is limited to well-organised data, usually captured in well-controlled conditions. We propose a novel general multilinear matrix decomposition method that discovers the multilinear structure of possibly incomplete sets of visual data in unsupervised setting. We demonstrate the applicability of the proposed method in several computer vision tasks, including Shape from Shading (SfS) (in the wild and with occlusion removal), expression transfer, and estimation of surface normals from images captured in the wild. Finally, leveraging the unsupervised multilinear method proposed as well as recent advances in deep learning, we propose a weakly supervised deep learning method for disentangling multiple latent factors of variation in face images captured in-the-wild. To this end, we propose a deep latent variable model, where we model the multiplicative interactions of multiple latent factors of variation explicitly as a multilinear structure. We demonstrate that the proposed approach indeed learns disentangled representations of facial expressions and pose, which can be used in various applications, including face editing, as well as 3D face reconstruction and classification of facial expression, identity and pose.

Data are collected at an unprecedented rate owing to the ubiquitous existence of sensors. Modeling such data, especially high-dimensional data such as images or videos, is of paramount importance. Deep generative models, that are implemented as multilayer neural networks, have demonstrated success in modeling distributions of high-dimensional data. In this thesis, we explore new structures, models and approximation methods for deep generative models. We initially focus on conditional data generation and study the robustness of the popular conditional Generative Adversarial Nets (cGAN). The model assumes a conditional input, e.g., a corrupted image, is available and we want to transform it into a high-dimensional output. However, our experiments illustrate that cGAN is not robust to noise. To mitigate that, we propose a new, augmented model, called RoCGAN, which leverages structure in the output space of the model. RoCGAN exhibits robust behavior even under intense noise. Sequentially, we turn our attention to the function approximation method used for implementing deep generative models, i.e., feedforward neural networks. Little attention has been given to the method of choice for function approximation. To that end, we model the data generator as a high-order polynomial. We conduct experiments with the polynomial generator using Generative Adversarial Nets (GANs). We illustrate that the polynomial expansion is a very expressive function approximator and natural signals, such as images, can be approximated without the use of activation functions (i.e., a cornerstone of the approximation with feedforward neural networks). The polynomial function approximation is generalized from modeling the generator of a GAN to modeling both generative and discriminative methods in diverse tasks, such as image or audio classification or representation learning in non-euclidean domains. Lastly, we focus on the type of generative model. Despite the stellar success of GANs, their complicated training dynamics, make the study of alternative models imperative. To that end, we introduce a decoder-only method that uses a polynomial expansion to approximate natural images.

Deep learning is now a predominant technique for most machine learning problems, especially in computer vision. With multiple layers, deep networks can extract hierarchical features from the images to solve highly non-linear problems. Currently, its development has been taken into many directions, such as modifying networks' architectures, using different losses during training, and adjusting the training strategies. Regarding facial analysis in imagery, it has been advantageous to learn statistical models for both appearances and shapes, which has not been fully explored in the context of deep learning. In this thesis, we examine how deep learning can be carefully designed to solve face analysis in both discriminative and generative settings more effectively. In particular, we incorporated the impressive feature extraction power from deep networks with spatial features provided by well-studied facial landmarks. Our proposed methods further improved the performance in each task, suggesting that deep learning can benefit from utilizing statistical prior associated with each problem.

Statistical Deformable Models (SDMs) represent a widely used collection of computer vision techniques. SDMs serve the purpose of modeling the deformation and variability of a specific class of objects, e.g. faces, ears, hands, human poses, etc. SDMs have shown success in many areas of visual image analysis. For example, in biometrics, SDMs are commonly used for recognition, classification, detection, and alignment. During the past few years, we have witnessed the development of many methodologies for building and fitting Statistical Deformable Models. The construction of accurate SDMs requires careful annotation of images with regards to a consistent set of landmarks. However, the manual annotation of a large number of images is a tedious, laborious and expensive procedure. Furthermore, for several deformable objects, e.g. the human body, it is difficult to define a consistent set of landmarks. Even though, the SDMs construction still suffers in terms of sparse landmark annotation. With sparse annotations, the shape deformation of any deformable objects is only captured where the key points are labeled, thus nuanced shape deformation is discarded. It is merely possible to accurately annotate dense shape deforestation manually. Fortunately, for the majority of objects, it is possible to extract the shape by object segmentation or even by shape drawing. In this thesis, we propose that it is possible to construct SDMs by putting objects shapes in dense correspondence. Such SDMs can be built with much less effort for a large battery of objects. Additionally, by sampling the dense model, a part-based SDM can be learned with its parts being in correspondence. We propose a framework to develop SDMs of human arms and legs, which can be used for the segmentation of the outline of the human body, as well as to provide better and more consistent annotations for articulated objects. Such a system can serve as an element of cascaded architectures that jointly localize landmarks and estimate dense correspondences. Also, we show that the obtained dense correspondence can act as a source of prior knowledge that complements and extends the pure landmark-level annotations, accelerating and improving the training of pose estimation networks. The thesis contributes towards fully automatic dense correspondence estimation. Such correspondence would benefit any system that requires nuance shape deformation. We are approaching a milestone that dense correspondence and poses for deformable objects can be estimated “in-the-wild” with minimal effort on labeling. The estimated correspondence is shown to be useful for various downstream tasks for a collection of objects.

During the last few years, we have witnessed tremendous advances in the field of 2D Deformable Models for the problem of landmark localization. These advances, which are mainly reported on the task of face alignment, have created two major and opposing families of methodologies. On the one hand, there are the generative Deformable Models that utilize a Newton-type optimization. This family of techniques has attracted extensive research effort during the last two decades, but has lately been criticized of achieving inaccurate performance. On the other hand, there is the currently predominant family of discriminative Deformable Models that treat the problem of landmark localization as a regression problem. These techniques commonly employ cascaded linear regression and have proved to be very accurate. In this thesis, we argue that even though generative Deformable Models are less accurate than discriminative, they are still very valuable for several tasks. In the first part of the thesis, we propose two novel generative Deformable Models. In the second part of the thesis, we show that the combination of generative and discriminative Deformable Models achieves state-of-the-art results on the tasks of (i) landmark localization and (ii) semi-supervised annotation of large visual data.

3D Morphable Models (3DMMs) are powerful statistical models of the 3D shape and texture of the human face, and algorithms based around them are presently state of the art for recovering the 3D geometry and appearance of a face from an image. Even so, the applications of such techniques are gated by a number of factors. Firstly, existing techniques for constructing 3DMMs require manual intervention, limiting the amount of training data that can be used. As 3DMM algorithms can only recover solutions present in the span of the statistical model used, this places a fundamental limit on reconstruction quality to the confines of face models trained from smaller cohorts of data. Secondly, existing approaches for 3D reconstruction from images using 3DMMs are either fragile to real-world "in-the-wild" image effects, or limited in their ability to recover person-specific detail. Finally, there is limited work on the recovery of 3D geometry from "in-the-wild" video sequences of an individual, a very common modality of data in 2017. This thesis takes steps forwards in all three of these domains. A novel pipeline for 3DMM construction is presented that is completely automated, allowing for the construction of large scale 3D Morphable Models for the first time. By using this approach on a new dataset of 10,000 facial scans, the Large Scale Facial Model (LSFM) is introduced, a new 3DMM which is shown to have far more representative power than previous facial models. It is demonstrated that this increased power translates into better performance in 3DMM applications, and a study is performed into the effects of demographic traits such as age, gender and ethnicity on 3D facial appearance. A new approach for fitting 3DMMs to images is developed which is capable of recovering shape detail without sacrificing robustness to "in-the-wild" effects. To achieve this, it is demonstrated that an "in-the-wild" texture model for a 3DMM can be learnt from a distribution of real world images. This new "in-the-wild" 3DMM benefits from a simple but effective cost function that can be reliably optimised even in the case of challenging facial images. Finally, videos of people are given specific treatment for the "in-the-wild" 3DMM, leading to an optimal solution for 3D face tracking in this common data modality.

This thesis investigates the use of robust deformable model for 3D face alignment and tracking. Our main objective is to establish accurate and reliable dense correspondence for 4D faces. Our contribution is two-fold. First, we build a robust face alignment framework to establish dense correspondence between 4D faces. Second, we develop 4DFAB, the first large scale high-resolution 4D face database for facial expression analysis and biometric. Our deformable 4D face alignment framework contains three parts: (1) robust 2D face alignment; (2) active non-rigid 3D face registration; (3) deformable 4D face tracking. For 2D face alignment, we start with the study of robust 2D and 3D geometry features (and their fusion) for Constrained Local Models (CLMs). We show that by leveraging robust features, CLMs can handle faces captured in controlled and uncontrolled environment. To exploit the discriminative power of CLMs, we propose an alignment framework based on the texture model of response maps. Under this framework, we devise two generative fitting methods (GFRM-Alt and GFRM-PO) and one part-based discriminative fitting method (DFRM), which achieve favorable performances in generic face alignment in-the-wild. Additionally, we implement a real-time face tracking software using DFRM. Next, we study the registration problem for high quality 3D facial scan. We build a part-based statistical face model and combine it with the non-rigid ICP. We name the method as Active Non-rigid ICP (ANICP). ANICP is integrated into a dynamic local fitting framework and produces accurate fitting. Note that DFRM is also used to provide better initialisation. Finally, we develop a dense 4D alignment framework that capitalises on GRMF-Alt and ANICP. This framework is employed to align faces from 4DFAB database, which contains over 1,800,000 meshes from 180 subjects captured in four different sessions during 5 years. As the subjects display both spontaneous and posed expressions, it can also be used for facial expression modeling and analysis. To demonstrate the usefulness of 4DFAB, we conduct extensive expression recognition, face recognition, speech recognition experiments. We also investigate, for the first time, the use of spontaneous 4D behaviour for biometric applications.

Component Analysis (CA) methods have been crucial contributors for the large success of machine learning over the past decades. Although predominately all CA methods are linear models, depending on the formulation of the optimisation problem one can derive vastly different results tailored to different tasks. Such linear methods have the natural advantage of intepretability as they can easily be reasoned about, but also they are easy to fit onto the available data. Unfortunately, as they are mostly linear models they can not describe complex data distributions such as in-the-wild images of faces or, videos, or even auditory signals. On the other hand deep learning methodologies have excelled in modelling highly non-linear relations between highly heterogenious data distributions. Nonetheless, they were mostly used as black-boxes and usually required orders of magnitude more training samples than their linear counterparts to achieve similar performance. In this thesis we will aim to combine the best of both worlds. That is to incorporate the power of neural networks with the statistical intuition and the specially crafted ideas of component analysis methods. The result methodologies will have a diverse application set, solving problems from the areas of face clustering, timeseries alignment, domain adaptation, and face alignment in a mainly unsupervised way.

Human faces are one of the most frequently captured objects in both videos and photographs due to their fundamental role in communication and social interactions. The variability of this facial imagery makes it difficult to automate the understanding of scenes containing faces under unconstrained conditions. For faces, recovering accurate dense 3D facial shape from images and videos enables much richer understanding of the human face and its interaction with the scene. In this thesis, we seek to extend the work in the area of dense 3D facial shape recovery under challenging unconstrained conditions. There are a wealth of ways to recover 3D shape from images and videos, all of which make specific assumptions about the relationship between the individual images and the construction of the scene. Given this broad selection of methods available, we examine three different scenarios for dense 3D facial shape recovery: i) recovery from a single image, ii) recovery from an unconstrained image collection without any explicit 3D shape priors and iii) recovery from a video sequence. We focus on these three cases and show how facial priors can be introduced to tackle the dense 3D facial surface recovery problem. We propose to investigate the use of shading constraints for dense shape recovery from unconstrained images. Given the challenging nature of these images, the introduction of priors greatly improves performance over the generic shape-from-shading literature. However, the introduction of explicit priors comes with a further problem, that of correspondence. That is, recovering the relationship between pixels in the image and the structure of our model. For this reason, we also investigate the importance of finding dense correspondences between facial images. We show that it is possible to recover plausible dense 3D facial surfaces under a variety of different input conditions.

This thesis studies component analysis techniques when complex-valued data arise. Until recently, the usual way to apply these techniques to complex-valued data has been either by splitting the complex-valued data into their real and imaginary parts and afterwards use the techniques with these real data or by applying directly the techniques in the complex domain with some mild assumptions. We have focused in bridging the gap between the two alternatives and shown how to work natively with complex data. To this end, we introduce the use of the quite recently proposed widely linear model on component analysis and we compare it with the classical approaches. Applications and experiments for these methods have been performed for the case of computer vision problems such as face reconstruction, face recognition, expression recognition and video tracking. In order to decipher the whole approach in the complex domain, we present an introduction to the theory of complex calculus, where the concepts of widely linear transformations, augmented matrix algebra, Wirtinger and complex-valued matrix derivatives are illustrated showing how all these principles can be used. Also, we conduct an overview of the recent advances in the field of augmented statistics and widely linear modeling. The theory of complex-valued kernels and its usage on Principal Component Analysis (PCA) is analysed in depth and the widely linear version of PCA is presented as well as applications of the proposed method. Furthermore, we examine shape representation in real and complex domain and we compare alternative representations for computer vision tasks. Finally, we have worked towards the unification of component analysis methods in a least-square framework and we look for similarities and differences between the circular and widely linear model.

This thesis presents a detailed and complete study of compositional gradient descent (CGD) algorithms for fitting active appearance models (AAM) and advances the state-of-the-art in generative AAM fitting by incorporating: (i) novel robust texture representations; (ii) novel cost functions and compositional types; and (iii) combined fitting approaches with complementary deformable models; into the original CGD framework. In particular, a robust texture representation based on image gradient orientations is used to define a new type of generative deformable model that generalizes well to variations in identity, pose, expression, illumination and occlusions and that can be fitted to images using standard CGD algorithms. Moreover, a novel Bayesian formulation of the AAM fitting problem, which can be interpreted as a probabilistic generalization of the well-known project-out inverse compositional (PIC) algorithm, is proposed along with two new types of composition, asymmetric and bidirectional, that lead to better convergent and more robust CGD fitting algorithms. At the same time, interesting insights into existent strategies used to derive fast and exact simultaneous CGD algorithms are provided by reinterpreting them as direct applications of the Schur complement and the Wiberg method. Finally, CGD algorithms are combined with similar generative fitting techniques for constrained local models (CLM) to create a unified probabilistic fitting framework that combines the strengths of both models (AAM and CLM) and produces state-of-the-art results on the problem of non-rigid face alignment in the wild.

The recent spark in machine learning and computer vision methods requiring increasingly larger datasets has motivated the introduction of optimisation algorithms specifically tailored to solve very large problems within practical time constraints. This demand in algorithms challenges the practicability of state of the art methods requiring new approaches that can take advantage of not only the problem’s mathematical structure, but also its data structure. Fortunately, such structure is present in many computer vision applications, where the problems can be modelled with varying degrees of fidelity. This structure suggests using multiscale models and thus multilevel algorithms. The objective of this thesis is to develop, implement and test provably convergent multilevel optimisation algorithms for convex composite optimisation problems in general and its applications in computer vision in particular. Our first multilevel algorithm solves convex composite optimisation problem and it is most efficient particularly for the robust facial recognition task. The method uses concepts from proximal gradient, mirror descent and multilevel optimisation algorithms, thus we call it multilevel accelerated gradient mirror descent algorithm (MAGMA). We first show that MAGMA has the same theoretical convergence rate as the state of the art first order methods and has much lower per iteration complexity. Then we demonstrate its practical advantage on many facial recognition problems. The second part of the thesis introduces new multilevel procedure most appropriate for the robust PCA problems requiring iterative SVD computations. We propose to exploit the multiscale structure present in these problems by constructing lower dimensional matrices and use its singular values for each iteration of the optimisation procedure. We implement this approach on three different optimisation algorithms - inexact ALM, Frank-Wolfe Thresholding and non-convex alternating projections. In this case as well we show that these multilevel algorithms converge (to an exact or approximate) solution with the same convergence rate as their standard counterparts and test all three methods on numerous synthetic and real life problems demonstrating that the multilevel algorithms are not only much faster, but also solve problems that often cannot be solved by their standard counterparts.

The rapid development of Augmented Reality (AR) and Virtual Reality (VR) applications over the past years has created the need to quickly and accurately scan the real world to populate immersive, realistic virtual environments for the end user to enjoy. While geometry processing has already gone a long way towards that goal, with self-contained solutions commercially available for on-site acquisition of large scale 3D models, capturing the appearance of the materials that compose those models remains an open problem in general uncontrolled environments. The appearance of a material is indeed a complex function of its geometry, intrinsic physical properties and furthermore depends on the illumination conditions in which it is observed, thus traditionally limiting the scope of reflectometry to highly controlled lighting conditions in a laboratory setup. With the rapid development of digital photography, especially on mobile devices, a new trend in the appearance modelling community has emerged, that investigates novel acquisition methods and algorithms to relax the hard constraints imposed by laboratory-like setups, for easy use by digital artists. While arguably not as accurate, we demonstrate the ability of such self-contained methods to enable quick and easy solutions for on-site reflectometry, able to produce compelling, photo-realistic imagery. In particular, this dissertation investigates novel methods for on-site acquisition of surface reflectance based on off-the-shelf, commodity hardware. We successfully demonstrate how a mobile device can be utilised to capture high quality reflectance maps of spatially-varying planar surfaces in general indoor lighting conditions. We further present a novel methodology for the acquisition of highly detailed reflectance maps of permanent on-site, outdoor surfaces by exploiting polarisation from reflection under natural illumination. We demonstrate the versatility of the presented approaches by scanning various surfaces from the real world and show good qualitative and quantitative agreement with existing methods for appearance acquisition employing controlled or semi-controlled illumination setups.

In many areas of science and technology data describing a phenomenon or a system of interest and its individual components can be obtained from different types of instruments, using different acquisition techniques and experimental setups. Furthermore, many systems can be characterized by different data sets obtained by observing the system from different perspectives, under different conditions, or at different points in time. For example, in biology and biomedicine, rapid technological advances have led to the production of such heterogeneous and multimodal data, and enabled construction of complex networks with various types of interactions between diverse biological entities. Conventional network data analysis methods were shown to be limited in dealing with such multi-relational network data, as they are designed specifically for analysis of single-relational networks. Therefore, in this Thesis, we propose novel integrative methods that can collectively mine multiple types of biological networks and create more accurate integrative models capable of producing more holistic, systems-level biological insights. The proposed methods are based on Non-negative Matrix Factorization, a dimensionality reduction technique, that is further extended for learning and modelling dependencies between large-scale heterogeneous and multimodal network data. Our methods have a wide range of applications of which we focus on those from systems biology and biomedicine, including protein function prediction, multiple network alignment, extraction of composite functional modules from multimodal biological networks, and problems in precision medicine, including patient strati cation, drug repurposing and cancer driver gene prediction. In each of these applications we show that our methods outperform existing methods, provide biological insights and generate novel hypothesis that could guide future research and biological experiments.

Automatic analysis of facial images is a problem of paramount importance due to its application in numerous real world scenarios including security, entertainment, medicine, health care, multimedia, human-computer, and human-robot interactions. Arguably the most important step of an automatic face analysis system is the localization of the facial landmarks. This is due to the fact that it has a crucial impact on the robustness and accuracy of the designed system. Facial landmarks localization is a very challenging Computer Vision problem, since the face is a highly deformable object and its appearance drastically changes under different poses, expressions, and illuminations conditions. Recently, Computer Vision has witnessed great research advance towards automatic facial landmarks localization. Numerous methodologies have been proposed during the last few years that achieve accurate and efficient performance. The most successful methods are based on statistical deformable models. Developing powerful facial deformable models requires massive, annotated facial databases on which techniques can be trained, validated and tested. The past twenty years the research community has collected and annotated a number of facial databases captured under both constrained and unconstrained (in-the-wild) conditions. However, the existing facial databases cannot be utilised for training the aforementioned models due to several limitations of the provided annotations. More specifically, most databases have been annotated using different mark-ups, and in most cases, the accuracy of the provided annotations is low. Additional to the aforementioned problems the use of different training/testing sets and different error metrics makes the fair comparison between the existing methodologies almost infeasible. In this Thesis, we first aim to overcome the aforementioned problems by (a) proposing a semi-automatic annotation technique that was employed to re-annotate most existing facial databases under a unified protocol, and (b) presenting the 300 Faces In-The-Wild Challenge (300-W), the first facial landmark localization challenge that was organized twice, in 2013 and 2015. This is the first effort towards a unified annotation scheme of massive databases and a fair experimental comparison of existing facial landmarks localization systems. By tracking the published papers in recent Computer Vision conferences it can be seen that the produced annotations allowed the researchers to propose powerful generic facial deformable models for facial landmarks localization in still images. Nevertheless, when it comes to applications that require perfect facial landmarks localization and tracking accuracy, such as the analysis of human facial behaviour and facial motion capture, generic facial deformable models could be insufficient. In this case, person-specific facial deformable models are mainly employed, requiring manual annotation of facial landmarks for each person and subsequently person-specific training. In this Thesis, a novel method for the automatic construction of person-specific facial deformable models is proposed. To this end, an orthonormal subspace which is suitable for facial image reconstruction is learned. Next, to correct the erroneous fittings produced by a generic facial deformable model, image congealing (i.e., ensemble image alignment) is performed by employing only the learned orthonormal subspace. The image congealing problem is solved by formulating a suitable sparsity regularized rank minimization problem. After correcting the fittings, the next step is to construct the person-specific facial deformable model which could be further used to localize or track the facial landmarks in images that depict the same subject. This consists another contribution of this Thesis. After applying a generic or person-specific facial deformable model into still or a sequence of facial images, the next step of an automatic face analysis system is to remove the pose effect from the faces. To do that, landmark points-driven normalization (i.e., warping) of the faces into a common frame (e.g., frontal-view frame) is performed. However, most face normalization (pose correction) methods can be greatly affected from huge poses, illumination variations, occlusions and bad localization of the facial landmarks. A final, significant contribution of this Thesis is the development of a novel method, robust to aforementioned problems, for joint face frontalization (i.e., pose correction) and facial landmarks localization. Unlike the state-of-the-art methods for facial landmarks localization and pose correction, where large amount of manually annotated images or 3D facial models are required, the proposed method relies on a small set of frontal images only. By observing that the frontal facial image of both humans and animals, is the one having the minimum rank of all different poses, a model which is able to jointly recover the frontalized version of the face as well as the facial landmarks is devised. Therefore, we solve the optimization problem concerning minimization of the nuclear norm and the matrix ell_1 norm accounting for occlusions. This method is assessed for frontal view reconstruction of human and animal faces, landmark localization, pose-invariant face recognition, face verification in unconstrained conditions, and video in-painting by conducting experiment on nine databases. The experimental results demonstrate the effectiveness of the proposed method in comparison to the state-of-the-art methods for the target problems.

Facial expressions are one of the most important means for communication of emotions and meaning. They are used to clarify and give emphasis, to express intentions, and form a crucial part of any human interaction. The ability to automatically recognise and analyse expressions could therefore prove to be vital in human behaviour understanding, which has applications in a number of areas such as psychology, medicine and security. 3D and 4D (3D+time) facial expression analysis is an expanding field, providing the ability to deal with problems inherent to 2D images, such as out-of-plane motion, head pose, and lighting and illumination issues. Analysis of data of this kind requires extending successful approaches applied to the 2D problem, as well as the development of new techniques. The introduction of recent new databases containing appropriate expression data, recorded in 3D or 4D, has allowed research into this exciting area for the first time. This thesis develops a number of techniques, both in 2D and 3D, that build towards a complete system for analysis of 4D expressions. Suitable feature types, designed by employing binary pattern methods, are developed for analysis of 3D facial geometry data. The full dynamics of 4D expressions are modelled, through a system reliant on motion-based features, to demonstrate how the different components of the expression (neutral-onset-apex-offset) can be distinguished and harnessed. Further, the spatial structure of expressions is harnessed to improve expression component intensity estimation in 2D videos. Finally, it is discussed how this latter step could be extended to 3D facial expression analysis, and also combined with temporal analysis. Thus, it is demonstrated that both spatial and temporal information, when combined with appropriate 3D features, is critical in analysis of 4D expression data.

Wearable electroencephalography (EEG) is a technology that is revolutionising the longitudinal monitoring of neurological and mental disorders, improving the quality of life of patients and accelerating the relevant research. As sleep disorders and other conditions related to sleep quality affect a large part of the population, monitoring sleep at home, over extended periods of time could have significant impact on the quality of life of people who suffer from these conditions. Annotating the sleep architecture of patients, known as sleep stage scoring, is an expensive and time-consuming process that cannot scale to a large number of people. Using wearable EEG and automating sleep stage scoring is a potential solution to this problem. In this thesis, we propose and evaluate two deep learning algorithms for automated sleep stage scoring using a single channel of EEG. In our first method, we use time-frequency analysis for extracting features that closely follow the guidelines that human experts follow, combined with an ensemble of stacked sparse autoencoders as our classification algorithm. In our second method, we propose a convolutional neural network (CNN) architecture for automatically learning filters that are specific to the problem of sleep stage scoring. We achieved state-of-the-art results (mean F1-score 84%; range 82-86%) with our first method and comparably good results with the second (mean F1-score 81%; range 79-83%). Both our methods effectively account for the skewed performance that is usually found in the literature due to sleep stage duration imbalance. We propose a filter analysis and visualisation methodology for CNNs to understand the filters that CNNs learn. Our results indicate that our CNN was able to robustly learn filters that closely follow the sleep scoring guidelines.