Search

Sharing facial pictures through online services, especially on social networks, has become a common habit for thousands of users. This practice hides a possible threat to privacy: the owners of such services, as well as malicious users, could automatically extract information from faces using modern and effective neural networks. In this article, we propose the harmless use of adversarial attacks, i.e., variations of images that are almost imperceptible to the human eye and that are typically generated with the malicious purpose to mislead Convolutional Neural Networks (CNNs). Such attacks have been instead adopted to (1) obfuscate soft biometrics (gender, age, ethnicity) but (2) without degrading the quality of the face images posted online. We achieve the above-mentioned two conflicting goals by modifying the implementations of four of the most popular adversarial attacks, namely FGSM, PGD, DeepFool, and C&W, in order to constrain the average amount of noise they generate on the image and the maximum perturbation they add on the single pixel. We demonstrate, in an experimental framework including three popular CNNs, namely VGG16, SENet, and MobileNetV3, that the considered obfuscation method, which requires at most 4 seconds for each image, is effective not only when we have a complete knowledge of the neural network that extracts the soft biometrics (white box attacks) but also when the adversarial attacks are generated in a more realistic black box scenario. Finally, we prove that an opponent can implement defense techniques to partially reduce the effect of the obfuscation, but substantially paying in terms of accuracy over clean images; this result, confirmed by the experiments carried out with three popular defense methods, namely adversarial training, denoising autoencoder, and Kullback-Leibler autoencoder, shows that it is not convenient for the opponent to defend himself and that the proposed approach is robust to defenses. [ABSTRACT FROM AUTHOR]

Facial landmarks detection is an essential step in many face analysis applications for ambient understanding (people, scenes) and for dynamically adapting the interaction with humans and environment. The current methods have difficulties with real-world images. This paper proposes a simple and effective method to detect the essential points in human faces. The proposed method comprises a two-stage coordinated regression deep convolutional neural network (CR-CNN) with a heatmap coupling module to convert the detected facial landmarks of the first stage into a Gaussian heatmap. To take advantage of the prior stage knowledge, the generated heatmap is concatenated with the original image of the input face and entered into the network in the second stage. The two-stage implementation based on CR-CNN has same layers structure to simplify the design and complexity. The L 1 loss function is used for each stage and the total loss equals the sum of the two loss functions from both stages. Comprehensive experiments are conducted to evaluate the proposed method on three common challenging facial landmark datasets, namely AFLW, 300W, and WFLW. The proposed method achieves normalized mean error (NME) of 1.56% on the AFLW, 4.20% on the 300W, and 5.53% on the WFLW datasets. Moreover, the execution time of the proposed two-stage CR-HC is calculated as 3.33 ms. The obtained results show the robustness and outstanding performance of the proposed method over some of the state-of-the-art methods. The source code is provided as an open repository to the community for further research activities. [ABSTRACT FROM AUTHOR]

Abstract Facial beauty analysis is an important topic in human society. It may be used as a guidance for face beautification applications such as cosmetic surgery. Deep neural networks (DNNs) have recently been adopted for facial beauty analysis and have achieved remarkable performance. However, most existing DNN‐based models regard facial beauty analysis as a normal classification task. They ignore important prior knowledge in traditional machine learning models which illustrate the significant contribution of the geometric features in facial beauty analysis. To be specific, landmarks of the whole face and facial organs are introduced to extract geometric features to make the decision. Inspired by this, we introduce a novel dual‐branch network for facial beauty analysis: one branch takes the Swin Transformer as the backbone to model the full face and global patterns, and another branch focuses on the masked facial organs with the residual network to model the local patterns of certain facial parts. Additionally, the designed multi‐scale feature fusion module can further facilitate our network to learn complementary semantic information between the two branches. In model optimisation, we propose a hybrid loss function, where especially geometric regulation is introduced by regressing the facial landmarks and it can force the extracted features to convey facial geometric features. Experiments performed on the SCUT‐FBP5500 dataset and the SCUT‐FBP dataset demonstrate that our model outperforms the state‐of‐the‐art convolutional neural networks models, which proves the effectiveness of the proposed geometric regularisation and dual‐branch structure with the hybrid network. To the best of our knowledge, this is the first study to introduce a Vision Transformer into the facial beauty analysis task.

Facial beauty analysis is an important topic in human society. It may be used as a guidance for face beautification applications such as cosmetic surgery. Deep neural networks (DNNs) have recently been adopted for facial beauty analysis and have achieved remarkable performance. However, most existing DNN‐based models regard facial beauty analysis as a normal classification task. They ignore important prior knowledge in traditional machine learning models which illustrate the significant contribution of the geometric features in facial beauty analysis. To be specific, landmarks of the whole face and facial organs are introduced to extract geometric features to make the decision. Inspired by this, we introduce a novel dual‐branch network for facial beauty analysis: one branch takes the Swin Transformer as the backbone to model the full face and global patterns, and another branch focuses on the masked facial organs with the residual network to model the local patterns of certain facial parts. Additionally, the designed multi‐scale feature fusion module can further facilitate our network to learn complementary semantic information between the two branches. In model optimisation, we propose a hybrid loss function, where especially geometric regulation is introduced by regressing the facial landmarks and it can force the extracted features to convey facial geometric features. Experiments performed on the SCUT‐FBP5500 dataset and the SCUT‐FBP dataset demonstrate that our model outperforms the state‐of‐the‐art convolutional neural networks models, which proves the effectiveness of the proposed geometric regularisation and dual‐branch structure with the hybrid network. To the best of our knowledge, this is the first study to introduce a Vision Transformer into the facial beauty analysis task. [ABSTRACT FROM AUTHOR]

Abstract How to represent a human face pattern? While it is presented in a continuous way in human visual system, computers often store and process it in a discrete manner with 2D arrays of pixels. The authors attempt to learn a continuous surface representation for face image with explicit function. First, an explicit model (EmFace) for human face representation is proposed in the form of a finite sum of mathematical terms, where each term is an analytic function element. Further, to estimate the unknown parameters of EmFace, a novel neural network, EmNet, is designed with an encoder‐decoder structure and trained from massive face images, where the encoder is defined by a deep convolutional neural network and the decoder is an explicit mathematical expression of EmFace. The authors demonstrate that our EmFace represents face image more accurate than the comparison method, with an average mean square error of 0.000888, 0.000936, 0.000953 on LFW, IARPA Janus Benchmark‐B, and IJB‐C datasets. Visualisation results show that, EmFace has a higher representation performance on faces with various expressions, postures, and other factors. Furthermore, EmFace achieves reasonable performance on several face image processing tasks, including face image restoration, denoising, and transformation.

How to represent a human face pattern? While it is presented in a continuous way in human visual system, computers often store and process it in a discrete manner with 2D arrays of pixels. The authors attempt to learn a continuous surface representation for face image with explicit function. First, an explicit model (EmFace) for human face representation is proposed in the form of a finite sum of mathematical terms, where each term is an analytic function element. Further, to estimate the unknown parameters of EmFace, a novel neural network, EmNet, is designed with an encoder‐decoder structure and trained from massive face images, where the encoder is defined by a deep convolutional neural network and the decoder is an explicit mathematical expression of EmFace. The authors demonstrate that our EmFace represents face image more accurate than the comparison method, with an average mean square error of 0.000888, 0.000936, 0.000953 on LFW, IARPA Janus Benchmark‐B, and IJB‐C datasets. Visualisation results show that, EmFace has a higher representation performance on faces with various expressions, postures, and other factors. Furthermore, EmFace achieves reasonable performance on several face image processing tasks, including face image restoration, denoising, and transformation. [ABSTRACT FROM AUTHOR]

In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic. • A model capable of automatic face parts re-arrangement without alignment constraints. • An alternative way of disentangling face parts for semantic image synthesis. • The possibility of enriching SOTA methods for automatic mask-to-RGB face synthesis. [ABSTRACT FROM AUTHOR]

The field of fine-grained semantic segmentation for a person's face and head, which includes identifying facial parts and head components, has made significant progress in recent years. However, this task remains challenging due to the difficulty of considering ambiguous occlusions and large pose variations. To address these difficulties, we propose a new framework called Mask-FPAN. Our framework includes a de-occlusion module that learns to parse occluded faces in a semi-supervised manner, taking into account face landmark localization, face occlusion estimations, and detected head poses. Additionally, we improve the robustness of 2D face parsing by combining a 3D morphable face model with the UV GAN. We also introduce two new datasets, named FaceOccMask-HQ and CelebAMaskOcc-HQ, to aid in face parsing work. Our proposed Mask-FPAN framework successfully addresses the challenge of face parsing in the wild and achieves significant performance improvements, with a mIoU increase from 0.7353 to 0.9013 compared to the current state-of-the-art on challenging face datasets. [Display omitted] • Novel Mask-FPAN framework improves face parsing on challenging datasets. • Occ-Autoencoders enhance occlusion handling and enable semi-supervised learning. • 3D morphable model + UV GAN boost robustness, enhancing face parsing. • Introducing high-quality FaceOccMask-HQ and CelebAMaskOcc-HQ datasets. • Sparse relationships addressed via integrated techniques for improved parsing. [ABSTRACT FROM AUTHOR]

Action Units (AU) are muscular activations used to describe facial expressions. Therefore accurate AU recognition unlocks unbiaised face representation which can improve face-based affective computing applications. From a learning standpoint AU detection is a multi-task problem with strong inter-task dependencies. To solve such problem, most approaches either rely on weight sharing, or add explicit dependency modelling by decomposing the joint task distribution using Bayes chain rule. If the latter strategy yields comprehensive inter-task relationships modelling, it requires imposing an arbitrary order into an unordered task set. Crucially, this ordering choice has been identified as a source of performance variations. In this paper, we present Multi-Order Network (MONET), a multi-task method with joint task order optimization. MONET uses a differentiable order selection to jointly learn task-wise modules with their optimal chaining order. Furthermore, we introduce warmup and order dropout to enhance order selection by encouraging order exploration. Experimentally, we first demonstrate MONET capacity to retrieve the optimal order in a toy environment. Second, we validate MONET architecture by showing that MONET outperforms existing multi-task baselines on multiple attribute detection problems chosen for their wide range of dependency settings. More importantly, we demonstrate that MONET significantly extends state-of-the-art performance in AU detection. [ABSTRACT FROM AUTHOR]

Face-based recognition methods usually need the image of the whole face to perform, but in some situations, only a fraction of the face is visible, for example wearing sunglasses or recently with the COVID pandemic we had to wear facial masks. In this work, we propose a network architecture made up of four deep learning streams that process each one a different face element, namely: mouth, nose, eyes, and eyebrows, followed by a feature merge layer. Therefore, the face is segmented into the part of interest by means of ROI masks to keep the same input size for the four network streams. The aim is to assess the capacity of different combinations of face elements in recognizing the subject. The experiments were carried out on the Masked Face Recognition Database (M2FRED) which includes videos of 46 participants. The obtained results are 96% of recognition accuracy considering the four face elements; and 92%, 87%, and 63% of accuracy for the best combination of three, two, and one face elements respectively. • We propose a DL-based part by part analysis of the face. • We apply a multi-input CNN to recognize the faces. • We consider four different face parts: eye, eyebrows, mouth, and nose. • We conduct an ablation study to infer the relevance of each part in face recognition task. [ABSTRACT FROM AUTHOR]

Cephalometry analysis of facial soft tissue plays an important role for anthropologists in facial plastic surgeries. There are two important problems in the field of medical for facial anthropometry analysis. (1) Image calibration and its management during facial surgery operation by surgeons are difficult and time-consuming. (2) The manual analysis of facial cephalometry using mechanical tools such as a ruler and the caliper is highly error-inclined. A major disadvantage of manual measurement is that this method requires training and skill. To overcome the mentioned problems, this paper presents an automatic method for facial anthropometry analysis applicable for facial plastic surgeries based on orthogonal images. The proposed algorithm includes three main steps: (1) eye and mouth region detection from the frontal view, (2) facial contour extraction from the profile view, and (3) facial landmark detection. In this study, an optimized version of Fuzzy C-Means (FCM) clustering is presented using the Grey Wolf Optimization (GWO) for lip and facial skin segmentation. For facial landmark localization from the profile view, first, facial skin segmentation is done from frontal and profile views. Then, eye and mouth regions are detected from the frontal view. In facial orthogonal images, the Y coordinates of the feature points are the same, approximately. Finally, nine landmarks from the profile view are detected using the orthogonality feature. Experiment results show that the proposed algorithm is effective in the analysis of facial plastic surgeries. [ABSTRACT FROM AUTHOR]

The investigation of influences in artists' works has been a subject of interest for art historians for many years. Therefore, computational methods can provide a new perspective for identifying these influences' relationships. Indeed, several studies in computer science have proposed techniques to analyze similarities between paintings using various features. Faces are a crucial aspect of perception in art and have also been the focus of several studies in computational aesthetics. In our previous work, we proposed a method for analyzing artworks and evaluating the influence of artists. The present study improves upon the previous research by extending the analysis of influences considering second-degree influences between artists and the impact of geographic proximity, obtaining better results in terms of Recall than the previous work. In addition, we evaluated the capability of our method to detect work-to-work relationships between each pair of artworks by the artists, and we found plausible and interesting results, even though they have not yet been proven in the literature. By conducting further analysis of data extracted from the faces of works of art, the goal is to enhance the previous findings in the literature and foster further discussion and collaboration between the fields of art and computer science. The objective is not to provide a definitive answer to the question of influences but to stimulate further research in this area, pointing out new possibilities of influence and explanations about these influences. [Display omitted] • Methodology to suggest influences between artists based on the faces of paintings. • Uses groups of visual features to perform the analyses. • Proposes a way to identify influences using second-degree relationships and location. • Presents a work-by-work comparison in order to evaluate the method. [ABSTRACT FROM AUTHOR]

Face analysis is an essential topic in computer vision that dealing with human faces for recognition or prediction tasks. The face is one of the easiest ways to distinguish the identity people. Face recognition is a type of personal identification system that employs a person's personal traits to determine their identity. Human face recognition scheme generally consists of four steps, namely face detection, alignment, representation, and verification. In this paper, we propose to extract information from human face for several tasks based on recent advanced deep learning framework. The proposed approach outperforms the results in the state-of-the-art. [ABSTRACT FROM AUTHOR]

Context: The anthropometric facial clinical proportions are used in the field of orthodontics, maxillofacial and plastic surgery for aesthetic or abnormality corrections. There is lack of enough literature on the facial profiles of Indians. Aim: To assess correlations between facial parameters and stature of young Maharashtrian women by using anthropometry. Settings and Design: It is a cross-sectional observational pilot study at Maharashtra Institute of Dental Sciences & Research, after approval from the Institutional Ethical Committee. Methods and Material: The study included 15 students of 21-23 years age selected by simple randomisation. The facial parameters were measured by sliding vernier calipers after identifying facial landmarks by stickers. Facial height (FH) in thirds; upper FH (UFH), middle FH (MFH) and lower FH (LFH); facial width (FW) and stature or overall height (OH) were calculated to define average facial features. Statistical Analysis: Multiple pairwise statistics and simple linear regression analyses were done for various dependent variables. Results: The means of UFH, MFH, LFH and total facial heights (TFH) were found to be 5.2 ± 0.54, 5.35 ± 0.34, 5.16 ± 0.44 and 15.7 ± 0.98 cm, respectively. The TFH showed a moderate correlation with stature (P = 0.05, r = 0.64) and a strong correlation with lower lip length (P = 0.001, r = 0.78). Facial width showed a negative correlation with facial shape (P = 0.05). Conclusions: The selected sample showed the statistically insignificant difference between UFH, MFH and LFH indicating equitable distribution among Indian women of Maharashtrian origin of 21-23 year age group. Longer TFH is positively correlated with higher stature and longer lower lip length. [ABSTRACT FROM AUTHOR]

Besides the many advances made in the facial detection and recognition fields, face recognition applied to visual images (VIS-FR) has received increasing interest in recent years, especially in the field of communication, identity authentication, public safety and to address the risk of terrorism and crime. These systems however encounter important problems in the presence of variations in pose, expression, age, occlusion, disguise, and lighting as these factors significantly reduce the recognition accuracy. To prevent problems in the visible spectrum, several researchers have recommended the use of infrared images. This paper provides an updated overview of deep infrared (IR) approaches in face recognition (FR) and analysis. First, we present the most widely used databases, both public and private, and the various metrics and loss functions that have been proposed and used in deep infrared techniques. We then review deep face analysis and recognition/identification methods proposed in recent years. In this review, we show that infrared techniques have given interesting results for face recognition, solving some of the problems encountered with visible spectrum techniques. We finally identify some weaknesses of current infrared FR approaches as well as many future research directions to address the IR FR limitations.

Context: The anthropometric facial clinical proportions are used in the field of orthodontics, maxillofacial and plastic surgery for aesthetic or abnormality corrections. There is lack of enough literature on the facial profiles of Indians. Aim: To assess correlations between facial parameters and stature of young Maharashtrian women by using anthropometry. Settings and Design: It is a cross-sectional observational pilot study at Maharashtra Institute of Dental Sciences & Research, after approval from the Institutional Ethical Committee. Methods and Material: The study included 15 students of 21–23 years age selected by simple randomisation. The facial parameters were measured by sliding vernier calipers after identifying facial landmarks by stickers. Facial height (FH) in thirds; upper FH (UFH), middle FH (MFH) and lower FH (LFH); facial width (FW) and stature or overall height (OH) were calculated to define average facial features. Statistical Analysis: Multiple pairwise statistics and simple linear regression analyses were done for various dependent variables. Results: The means of UFH, MFH, LFH and total facial heights (TFH) were found to be 5.2 ± 0.54, 5.35 ± 0.34, 5.16 ± 0.44 and 15.7 ± 0.98 cm, respectively. The TFH showed a moderate correlation with stature (P ≤ 0.05, r = 0.64) and a strong correlation with lower lip length (P = 0.001, r = 0.78). Facial width showed a negative correlation with facial shape (P ≤ 0.05). Conclusions: The selected sample showed the statistically insignificant difference between UFH, MFH and LFH indicating equitable distribution among Indian women of Maharashtrian origin of 21–23 year age group. Longer TFH is positively correlated with higher stature and longer lower lip length.

In this study, we proposed a multitask network architecture for three attributes, landmark, head pose, and occlusion, from a face image. A 2-stacked hourglass with three task-specific heads is the proposed network architecture. We also designed three auxiliary components for the network. First is the feature pyramid fusion module, which plays a crucial role in facilitating contextual information from various receptive fields. Second is the interlevel occlusion-aware fusion module, which explicitly fuses intermediate occlusion prediction between subnetworks. The third is the gimbal-lock-free head pose head, which outputs a rotation matrix from a 6D rotation representation. We conducted an ablative study of these auxiliary components to determine their impacts on the network. Additionally, we introduced the landmark heatmap scaling approach to avoid falling local minima. We trained the proposed network with a 300W-LP dataset for landmark and head pose and a C-CM dataset for occlusion. Then, we fine-tuned the network using the 300W or WFLW dataset, instead of the 300W-LP dataset for the landmark task. This 2-stage training method contributes to enhancing the landmark detection accuracy and that of other tasks. In the experiments, we assessed the performance of the proposed network on eight test datasets using task-specific metrics. The results show that the proposed network achieved competitive performance across all the datasets and notably outperformed the state-of-the-art methods on AFLW2000 and Masked 300W datasets.

Facial landmark detection, a crucial aspect of face recognition, is widely used in various fields, such as facial surgeries, biometrics, and surveillance systems. With the advancement of affordable and capable 3D scanning technologies, research on automatically detecting facial landmarks on 3D models is gaining momentum. Utilizing the geometric properties of 3D facial models, researchers have developed algorithms for various landmarks with varying levels of accuracy. In this study, we reviewed existing literature and developed algorithms for thirty-eight landmarks using geometric properties and statistical information about facial measurements. The algorithms for thirty landmarks are original contributions to the literature. We provide the implementation of all the algorithms as open-source Python code, along with the pseudocode for both our algorithms and those found in the literature. To the best of our knowledge, this study covers the largest number of facial landmark detection algorithms based on the geometric properties of 3D models. This is the first study that provides the implementation of the algorithms along with detailed pseudocode. The results of the algorithms are presented by calculating the mean, median, standard deviation, minimum, and maximum of the errors and depicting the histogram for each landmark over a hundred 3D facial scans. The results show that geometric properties and statistics can be utilized to achieve more robust solutions for facial landmark detection.

Facial attributes are fundamental for studying deep structured information. Single-task face analysis reaches great performance, while analysis of multiple attributes meets challenges, including the network design and cross-dataset learning. In this paper, we propose cross-dataset face analysis based on multi-task learning (CFA-Net), which accomplishes landmark, head pose, age, gender, facial expression, and Action Unit (AU) analysis. Firstly, we balance between the shared and the task-specific structure to design an efficient and accurate network. To guarantee the excellent performance of each task, we utilize classification-based, regression-based, ranking-based, or deep label distribution learning-based methods to extract specific features for diverse tasks. Then, face analysis trained on a single dataset has strict requirements for this dataset. Even if this dataset currently meets the demand, the scalability is poor when tasks increase. Therefore, our training set is a mixture of multiple datasets, and each dataset covers one or several task related labels. Each sample possesses one or several tasks' labels, and we adopt a sample-dependent loss strategy, which only penalizes available ground truth. The proposed CFA-Net only occupies 1.58G GPU memory and costs 0.021s to address one image. In summary, the proposed CFA-Net behaves fast, occupies less memory, and performs well in every subtask, even better than those under single-task training. [ABSTRACT FROM AUTHOR]

Emotion recognition from face images is a challenging task that gained interest in recent years for its applications to business intelligence and social robotics. Researchers in computer vision and affective computing focused on optimizing the classification error on benchmark data sets, which do not extensively cover possible variations that face images may undergo in real environments. Following on investigations carried out in the field of object recognition, we evaluated the robustness of existing methods for emotion recognition when their input is subjected to corruptions caused by factors present in real-world scenarios. We constructed two data sets on top of the RAF-DB test set, named RAF-DB-C and RAF-DB-P, that contain images modified with 18 types of corruption and 10 of perturbation. We benchmarked existing networks (VGG, DenseNet, SENet and Xception) trained on the original images of RAF-DB and compared them with ARM, the current state-of-the-art method on the RAF-DB test set. We carried out an extensive study on the effects that modifications to the training data or network architecture have on the classification of corrupted and perturbed data. We observed a drop of recognition performance of ARM, with the classification error raising up to 200% of that achieved on the original RAF-DB test set. We demonstrate that the use of the AutoAugment data augmentation and an anti-aliasing filter within down-sampling layers provide existing networks with increased robustness to out-of-distribution variations, substantially reducing the error on corrupted inputs and outperforming ARM. We provide insights about the resilience of existing emotion recognition methods and an estimation of their performance in real scenarios. The processing time required by the modifications we investigated (35 ms in the worst case) supports their suitability for application in real-world scenarios. The RAF-DB-C and RAF-DB-P test sets, trained models and evaluation framework are available at https://github.com/MiviaLab/emotion-robustness. [ABSTRACT FROM AUTHOR]

Besides the many advances made in the facial detection and recognition fields, face recognition applied to visual images (VIS-FR) has received increasing interest in recent years, especially in the field of communication, identity authentication, public safety and to address the risk of terrorism and crime. These systems however encounter important problems in the presence of variations in pose, expression, age, occlusion, disguise, and lighting as these factors significantly reduce the recognition accuracy. To prevent problems in the visible spectrum, several researchers have recommended the use of infrared images. This paper provides an updated overview of deep infrared (IR) approaches in face recognition (FR) and analysis. First, we present the most widely used databases, both public and private, and the various metrics and loss functions that have been proposed and used in deep infrared techniques. We then review deep face analysis and recognition/identification methods proposed in recent years. In this review, we show that infrared techniques have given interesting results for face recognition, solving some of the problems encountered with visible spectrum techniques. We finally identify some weaknesses of current infrared FR approaches as well as many future research directions to address the IR FR limitations. [ABSTRACT FROM AUTHOR]

Heart rate is an important vital characteristic which indicates physical and mental health status. Typically heart rate measurement instruments require direct contact with the skin which is time-consuming and costly. Therefore, the study of non-contact heart rate measurement methods is of great importance. Based on the principles of photoelectric volumetric tracing, we use a computer device and camera to capture facial images, accurately detect face regions, and to detect multiple facial images using a multi-target tracking algorithm. Then after the regional segmentation of the facial image, the signal acquisition of the region of interest is further resolved. Finally, frequency detection of the collected Photoplethysmography (PPG) and Electrocardiography (ECG) signals is completed with peak detection, Fourier analysis, and a Wavelet filter. The experimental results show that the subject's heart rate can be detected quickly and accurately even when monitoring multiple facial targets simultaneously. [ABSTRACT FROM AUTHOR]

Deep learning (DL) models have been successfully applied in video-based affective computing, allowing, for instance, to recognize emotions and mood, or to estimate the intensity of pain or stress of individuals based on their facial expressions. Despite the recent advances with state-of-the-art DL models for spatio-temporal recognition of facial expressions associated with depressive behaviour, some key challenges remain in the cost-effective application of 3D-CNNs: (1) 3D convolutions usually employ structures with fixed temporal depth that decreases the potential to extract discriminative representations due to the usually small difference of spatio-temporal variations along different depression levels; and (2) the computational complexity of these models with consequent susceptibility to overfitting. To address these challenges, we propose a novel DL architecture called the Maximization and Differentiation Network (MDN) in order to effectively represent facial expression variations that are relevant for depression assessment. The MDN, operating without 3D convolutions, explores multiscale temporal information using a maximization block that captures smooth facial variations and a difference block that encodes sudden facial variations. Extensive experiments using our proposed MDN with models with 100 and 152 layers result in improved performance while reducing the number of parameters by more than $3\times$ 3 × when compared with 3D ResNet models. Our model also outperforms other 3D models and achieves state-of-the-art results for depression detection. Code available at: https://github.com/wheidima/MDN. [ABSTRACT FROM AUTHOR]

A number of modern techniques for neural network training and recognition enhancement are based on their structures' symmetry. Such approaches demonstrate impressive results, both for recognition practice, and for understanding of data transformation processes in various feature spaces. This survey examines symmetrical neural network architectures—Siamese and triplet. Among a wide range of tasks having various mathematical formulation areas, especially effective applications of symmetrical neural network architectures are revealed. We systematize and compare different architectures of symmetrical neural networks, identify genetic relationships between significant studies of different authors' groups, and discuss opportunities to improve the element base of such neural networks. Our survey builds bridges between a large number of isolated studies with significant practical results in the considered area of knowledge, so that the presented survey acquires additional relevance. [ABSTRACT FROM AUTHOR]

Introduction: The practicality of the idea whether the laughter‐involved large‐scale brain networks can be stimulated to remediate affective symptoms, namely depression, has remained elusive. Methods: In this study, 25 healthy individuals were tested through 21‐channel quantitative electroencephalography (qEEG) setup upon resting state and while submitted to standardized funny video clips (corated by two behavioral neuroscientists and a verified expert comedian, into neutral and mildly to highly funny). We evaluated the individuals' facial expressions against the valence and intensity of each stimulus through the Nuldos face analysis software. The study also employed an eye‐tracking setup to examine fixations, gaze, and saccadic movements upon each task. In addition, changes in polygraphic parameters were monitored upon resting state and exposure to clips using the 4‐channel Nexus polygraphy setup. Results: The happy facial expression analysis, as a function of rated funny clips, showed a significant difference against neutral videos (p < 0.001). In terms of the polygraphic changes, heart rate variability and the trapezius muscle surface electromyography measures were significantly higher upon exposure to funny vs. neutral videos (p < 0.5). The average pupil size and fixation drifts were significantly higher and lower, respectively, upon exposure to funny videos (p < 0.01). The qEEG data revealed the highest current source density (CSD) for the alpha frequency band localized in the left frontotemporal network (FTN) upon exposure to funny clips. Additionally, left FTN acquired the highest value for theta coherence z‐score, while the beta CSD predominantly fell upon the salience network (SN). Conclusions: These preliminary data support the notion that left FTN may be targeted as a cortical hub for noninvasive neuromodulation as a single or adjunct therapy in remediating affective disorders in the clinical setting. Further studies are needed to test the hypotheses derived from the present report. [ABSTRACT FROM AUTHOR]

Recently, deep learning models have been successfully employed in many video-based affective computing applications (e.g., detecting pain, stress, and Alzheimer’s disease). One key application is automatic depression recognition – recognition of facial expressions associated with depressive behaviour. State-of-the-art deep learning algorithms to recognize depression typically explore spatial and temporal information individually, by using 2D convolutional neural networks (CNNs) to analyze appearance information, and then by either mapping facial feature variations or averaging the depression level over video frames. This approach has limitations in terms of its ability to represent dynamic information that can help to accurately discriminate between depression levels. In contrast, models based on 3D CNNs allow to directly encode the spatio-temporal relationships, although these models rely on temporal information with fixed range and single receptive field. This approach limits the ability to capture variations of facial expression with diverse ranges, and the exploitation of diverse facial areas. In this article, a novel 3D CNN architecture – the Multiscale Spatiotemporal Network (MSN) – is introduced to effectively represent facial information related to depressive behaviours from videos. The basic structure of the model is composed of parallel convolutional layers with different temporal depths and sizes of receptive field, which allows the MSN to explore a wide range of spatio-temporal variations in facial expressions. Experimental results on two benchmark datasets show that our MSN architecture is effective, outperforming state-of-the-art methods in automatic depression recognition. [ABSTRACT FROM AUTHOR]

Abstract Introduction The practicality of the idea whether the laughter‐involved large‐scale brain networks can be stimulated to remediate affective symptoms, namely depression, has remained elusive. Methods In this study, 25 healthy individuals were tested through 21‐channel quantitative electroencephalography (qEEG) setup upon resting state and while submitted to standardized funny video clips (corated by two behavioral neuroscientists and a verified expert comedian, into neutral and mildly to highly funny). We evaluated the individuals’ facial expressions against the valence and intensity of each stimulus through the Nuldos face analysis software. The study also employed an eye‐tracking setup to examine fixations, gaze, and saccadic movements upon each task. In addition, changes in polygraphic parameters were monitored upon resting state and exposure to clips using the 4‐channel Nexus polygraphy setup. Results The happy facial expression analysis, as a function of rated funny clips, showed a significant difference against neutral videos (p < 0.001). In terms of the polygraphic changes, heart rate variability and the trapezius muscle surface electromyography measures were significantly higher upon exposure to funny vs. neutral videos (p < 0.5). The average pupil size and fixation drifts were significantly higher and lower, respectively, upon exposure to funny videos (p < 0.01). The qEEG data revealed the highest current source density (CSD) for the alpha frequency band localized in the left frontotemporal network (FTN) upon exposure to funny clips. Additionally, left FTN acquired the highest value for theta coherence z‐score, while the beta CSD predominantly fell upon the salience network (SN). Conclusions These preliminary data support the notion that left FTN may be targeted as a cortical hub for noninvasive neuromodulation as a single or adjunct therapy in remediating affective disorders in the clinical setting. Further studies are needed to test the hypotheses derived from the present report.

Human face image analysis using machine learning is an important element in computer vision. The human face image conveys information such as age, gender, identity, emotion, race, and attractiveness to both human and computer systems. Over the last ten years, face analysis methods using machine learning have received immense attention due to their diverse applications in various tasks. Although several methods have been reported in the last ten years, face image analysis still represents a complicated challenge, particularly for images obtained from 'in the wild' conditions. This survey paper presents a comprehensive review focusing on methods in both controlled and uncontrolled conditions. Our work illustrates both merits and demerits of each method previously proposed, starting from seminal works on face image analysis and ending with the latest ideas exploiting deep learning frameworks. We show a comparison of the performance of the previous methods on standard datasets and also present some promising future directions on the topic. [ABSTRACT FROM AUTHOR]

With the increasing development of advanced unmanned aerial vehicles (UAVs), communication between operators and these intelligent systems is becoming more stressful. For the safety of UAV flights, automatic psychological stress detection is becoming a key research topic for successful missions. Stress can be reliably estimated via some biological markers which are not appropriate in many cases of human-machine-interaction setups. In this article, we propose a non-intrusive deep learning-based stress level estimation approach. The goal is to identify the region where the operator's emotional state projects in the space defined by the latent dimensional emotions of arousal and valence since the stress region is well delimited in this space. The proposed multimodal approach uses sequential temporal CNN and LSTM with an Attention Weighted Average layer in the vision modality. As a second modality, we investigate local and global descriptors such as Mel-frequency cepstral coefficients, i-vector embeddings as well as Fisher-vector encodings. The multimodal-fusion approach uses a strategy referred to as “late-fusion” that involves the combination of unimodal model outputs as inputs of the decision engine. Since we have to deal with more naturalistic behavior in operator-machine interaction contexts, the One minute Gradual Emotion Challenge dataset was used for predictive model validation. [ABSTRACT FROM AUTHOR]

Facial skin diseases occur due to multiple reasons. They may have different or similar phenotypic signs and may psychologically and physically impact the affected person. Therefore, early detection, diagnosis, and prognosis of such skin diseases are prerequisite for proper treatment. In particular, an artificial-intelligence-based system helps dermatologists to identify facial skin diseases, even remotely. Within this context, the purpose of this research is to develop an algorithm to identify from a single face image, potential facial diseases. Only facial phenotypes are used regardless of the condition of acquisition (face pose, illumination, image resolution, etc.). Moreover, no segmentation of region of interests (ROIs) is required as commonly considered in the literature. Technically speaking, a calibrated CNN-based deep neural architecture facial skin diseases network (FSDNet) is proposed. It is a fine-tuned version of VGG 16 with modification of the architecture of the fully connected layer to be suitable for facial skin diseases identification. Due to the absence of any standard public dataset for the same, we created a database composed of 20000 images (with labeled pathologies) collected from different sources, which is used to train and validate our network. Our study achieves the identification of eight face skin pathologies, normal skin class, and no-face class with an accuracy of 97% [ABSTRACT FROM AUTHOR]

Recent work in unsupervised learning has focused on effcient inference and learning in latent variables models. Training these models by maximizing the evidence (marginal likelihood) is typically intractable. Thus, a common approximation is to maximize the Evidence Lower BOund (ELBO) instead. Variational autoencoders (VAE) are a powerful and widely-used class of generative models that optimize the ELBO effciently for large datasets. However, the VAE's default Gaussian choice for the prior imposes a strong constraint on its ability to represent the true posterior, thereby degrading overall performance. A Gaussian mixture model (GMM) would be a richer prior but cannot be handled effciently within the VAE framework because of the intractability of the Kullback-Leibler divergence for GMMs. We deviate from the common VAE framework in favor of one with an analytical solution for Gaussian mixture prior. To perform effcient inference for GMM priors, we introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. This new objective allows us to incorporate richer, multi-modal priors into the autoencoding framework. We provide empirical studies on a range of datasets and show that our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis. [ABSTRACT FROM AUTHOR]

A face does not only have rigid variations but also non-rigid distortions, which has influenced the performance of groupwise face alignment. A novel method for groupwise face alignment which considers both rigid variations of a face and non-rigid distortions was presented in the paper. The process for groupwise face alignment was divided into two stages, i.e. affine transformations and non-rigid distortions. At the stage of the affine transformations, the key points of a face were categorized into five groups and the affine transformations were used for each group of the key points. At the stage of the non-rigid distortions, a novel method was used for all of the key points in a face. Two stages were independent of each other, and moreover, iterations were made in each stage. Besides, the results from the stage of the affine transformations were used as the input of the non-rigid distortion stage. Experiments show that the method for groupwise face alignment in the paper is better than that only considering global affine variations, and is also better than that considering global affine variations and local non-rigid distortions. The method in the paper can be used as a novel method for groupwise face alignment.

Gender recognition has been among the most investigated problems in the last years; although several contributions have been proposed, gender recognition in unconstrained environments is still a challenging problem and a definitive solution has not been found yet. Furthermore, Deep Convolutional Neural Networks (DCNNs) achieve very interesting performance, but they typically require a huge amount of computational resources (CPU, GPU, RAM, storage), that are not always available in real systems, due to their cost or to specific application constraints (when the application needs to be installed directly on board of low-power smart cameras, e.g. for digital signage). In the latest years the Machine Learning community developed an interest towards optimizing the efficiency of Deep Learning solutions, in order to make them portable and widespread. In this work we propose a compact DCNN architecture for Gender Recognition from face images that achieves approximately state of the art accuracy at a highly reduced computational cost (almost five times). We also perform a sensitivity analysis in order to show how some changes in the architecture of the network can influence the tradeoff between accuracy and speed. In addition, we compare our optimized architecture with popular efficient CNNs on various common benchmark dataset, widely adopted in the scientific community, namely LFW, MIVIA-Gender, IMDB-WIKI and Adience, demonstrating the effectiveness of the proposed solution.

Researches on face alignment have made great progress, which benefits from the use of prior information and auxiliary models. However, that information lacks in a single face image has always affected the further development of these researches. The methods considering multiple face images provide a feasible way to solve the problem undoubtedly. Joint alignment where multiple face images are considered was presented in the paper. Face alignment was used for each face, and joint face alignment was used for optimizing the alignment results of all faces further. During joint alignment, both rigid variations of faces and non-rigid distortions were considered, however, they were regarded as two independent stages. Joint face alignment was a process where optimization was performed iteratively. In each iteration, both rigid variations and non-rigid distortions were performed sequentially, and moreover, the results of rigid variations were used as input of non-rigid distortions. At the stage of rigid variations, the key points of a face were divided into five groups to reduce the effect of global constraints which was imposed by face shape. After several iterations, the optimal solution of joint alignment can be obtained. The experimental results show that the joint alignment can obtain the optimal results than joint alignment using phased global rigid variations and non-rigid distortions and that using iterative global rigid variations and non-rigid distortions, and it can be used as a novel method for joint alignment.