Search

Existing face anti-spoofing models using deep learning for multi-modality data suffer from low generalization in the case of using variety of presentation attacks such as 2D printing and high-precision 3D face masks. One of the main reasons is that the non-linearity of multi-spectral information used to preserve the intrinsic attributes between a real and a fake face are not well extracted. To address this issue, we propose a multi-modility data based two-stage cascade framework for face anti-spoofing. The proposed framework has two advantages. Firstly, we design a two-stage cascade architecture that can selectively fuse low-level and high-level features from different modalities to improve feature representation. Secondly, we use multi-modality data to construct a distance-free spectral on RGB and infrared (IR) to augment the non-linearity of data. The presented data fusion strategy is different from popular fusion approaches, since it can strengthen discrimination ability of network models on physical attribute features than identity structure features under certain constraints. In addition, a multi-scale patch based weighted fine-tuning strategy is designed to learn each specific local face region. Experimental results show that the proposed framework achieves better performance than other state-of-the-art methods on both benchmark datasets and self-established datasets, especially on multi-material masks spoofing.

© 2021 The Authors. Many biomedical applications require accurate non-rigid image registration that can cope with complex deformations. However, popular diffeomorphic Demons registration algorithms suffer from difficulties for complex and serious distortions since they only use image greyscale and gradient information. To address these difficulties, a new diffeomorphic Demons registration algorithm is proposed using hierarchical neighbourhood spectral features namely HNSF Log-Demons in this paper. In view of three important properties of hierarchical neighbourhood spectral features based on line graph such as rotation invariance, invariance of linear changes of brightness, and robustness to noise, the hierarchical neighbourhood spectral features of a reference image and a moving image is first extracted and these novel spectral features are incorporated into the energy function of the diffeomorphic registration framework to improve the capability of capturing complex distortions. Secondly, the Nystr ö o ̈ m approximation based on random singular value decomposition is employed to effectively enhance the computational efficiency of HNSF Log-Demons. Finally, the hybrid multi-resolution strategy based on wavelet decomposition in the registration process is utilised to further improve the registration accuracy and efficiency. Experimental results show that the proposed HNSF Log-Demons not only effectively ensures the generation of smooth and reversible deformation field, but also achieves better performance than state-of-the-art algorithms. National Natural Science Foundation of China. Grant Numbers: 61762058, 61861024, 61871259; Natural Science Foundation of Gansu Province of China. Grant Number: 20JR5RA404; Natural Science Basic Research Program of Shaanxi. Grant Number: 2021JC-47.

Cosparse analysis model (CAM) provides a new signal processing paradigm for recovering cosparse signals with respect to a given analysis operator from the undersampled linear measurements in the context of emerging theory of compressed sensing (CS). The sparse analysis recovery/cosparse recovery is a key one brought up by this new paradigm. In this paper, we propose a new family of analysis pursuit algorithms for the sparse analysis recovery problem when the signals obey the cosparse analysis model, termed as iterative cosupport detection estimation (ICDE). ICDE is an algorithmic framework, which alternates between detecting a cosupport set of the unknown true signal and estimating the underlying signal by solving a truncated analysis pursuit problem on the detected cosupport. Further, we propose effective implementations of ICDE equipped with an efficient thresholding strategy for cosupport detection. Empirical performance comparisons show that ICDE is favorable in comparison with the state-of-the-art sparse analysis recovery algorithms. Source code of ICDE has been made publicly available on Github: https://github.com/songhp/ICDE. Beijing Natural Science Foundation (BNSF) under Grant No. 4194076, the Natural Science Foundation of Jiangsu Province under Grant No. BK20170558 and the China Scholarship Council (CSC, No. 202008320094).

Clustering algorithms by minimizing an objective function share a clear drawback of having to set the number of clusters manually. Although density peak clustering is able to find the number of clusters, it suffers from memory overflow when it is used for image segmentation because a moderate-size image usually includes a large number of pixels leading to a huge similarity matrix. To address this issue, here we proposed an automatic fuzzy clustering framework (AFCF) for image segmentation. The proposed framework has threefold contributions. First, the idea of superpixel is used for the density peak (DP) algorithm, which efficiently reduces the size of the similarity matrix and thus improves the computational efficiency of the DP algorithm. Second, we employ a density balance algorithm to obtain a robust decision-graph that helps the DP algorithm achieve fully automatic clustering. Finally, a fuzzy c-means clustering based on prior entropy is used in the framework to improve image segmentation results. Because the spatial neighboring information of both the pixels and membership are considered, the final segmentation result is improved effectively. Experiments show that the proposed framework not only achieves automatic image segmentation, but also provides better segmentation results than state-of-the-art algorithms. 10.13039/501100001809-National Natural Science Foundation of China; 10.13039/501100002858-China Postdoctoral Science Foundation

Although automatic fuzzy clustering framework (AFCF) based on improved density peak clustering is able to achieve automatic and efficient image segmentation, the framework suffers from two problems. The first one is that the adaptive morphological reconstruction (AMR) employed by the AFCF is easily influenced by the initial structuring element. The second one is that the improved density peak clustering using a density balance strategy is complex for finding potential clustering centers. To address these two problems, we propose a fast and automatic image segmentation algorithm using superpixel-based graph clustering (FAS-SGC). The proposed algorithm has two major contributions. First, the AMR based on regional minimum removal (AMR-RMR) is presented to improve the superpixel result generated by the AMR. The binary morphological reconstruction is performed on a regional minimum image, which overcomes the problem that the initial structuring element of the AMR is chosen empirically, since the geometrical information of images is effectively explored and utilized. Second, we use an eigenvalue gradient clustering (EGC) instead of improved density peak (DP) algorithms to obtain potential clustering centers, since the EGC is faster and requires fewer parameters than the DP algorithm. Experiments show that the proposed algorithm is able to achieve automatic image segmentation, providing better segmentation results while requiring less execution time than other state-of-the-art algorithms.

A great number of improved fuzzy c-means (FCM) clustering algorithms have been widely used for grayscale and color image segmentation. However, most of them are time-consuming and unable to provide desired segmentation results for color images due to two reasons. The first one is that the incorporation of local spatial information often causes a high computational complexity due to the repeated distance computation between clustering centers and pixels within a local neighboring window. The other one is that a regular neighboring window usually breaks up the real local spatial structure of images and thus leads to a poor segmentation. In this work, we propose a superpixel-based fast FCM clustering algorithm (SFFCM) that is significantly faster and more robust than state-of-the-art clustering algorithms for color image segmentation. To obtain better local spatial neighborhoods, we firstly define a multiscale morphological gradient reconstruction (MMGR) operation to obtain a superpixel image with accurate contour. In contrast to traditional neighboring window of fixed size and shape, the superpixel image provides better adaptive and irregular local spatial neighborhoods that are helpful for improving color image segmentation. Secondly, based on the obtained superpixel image, the original color image is simplified efficiently and its histogram is computed easily by counting the number of pixels in each region of the superpixel image. Finally, we implement FCM with histogram parameter on the superpixel image to obtain the final segmentation result. Experiments performed on synthetic images and real images demonstrate that the proposed algorithm provides better segmentation results and takes less time than state-of-the-art clustering algorithms for color image segmentation. China Postdoctoral Science Foundation; National Natural Science Foundation of China; National Science Foundation of Shanghai

As fuzzy c-means clustering (FCM) algorithm is sensitive to noise, local spatial information is often introduced to an objective function to improve the robustness of the FCM algorithm for image segmentation. However, the introduction of local spatial information often leads to a high computational complexity, arising out of an iterative calculation of the distance between pixels within local spatial neighbors and clustering centers. To address this issue, an improved FCM algorithm based on morphological reconstruction and membership filtering (FRFCM) that is significantly faster and more robust than FCM is proposed in this paper. First, the local spatial information of images is incorporated into FRFCM by introducing morphological reconstruction operation to guarantee noise-immunity and image detail-preservation. Second, the modification of membership partition, based on the distance between pixels within local spatial neighbors and clustering centers, is replaced by local membership filtering that depends only on the spatial neighbors of membership partition. Compared with state-of-the-art algorithms, the proposed FRFCM algorithm is simpler and significantly faster, since it is unnecessary to compute the distance between pixels within local spatial neighbors and clustering centers. In addition, it is efficient for noisy image segmentation because membership filtering are able to improve membership partition matrix efficiently. Experiments performed on synthetic and real-world images demonstrate that the proposed algorithm not only achieves better results, but also requires less time than the state-of-the-art algorithms for image segmentation.

Traditional fuzzy clustering algorithms suffer from two problems in image segmentations. One is that these algorithms are sensitive to outliers due to the non-sparsity of fuzzy memberships. The other is that these algorithms often cause image over-segmentation due to the loss of image local spatial information. To address these issues, we propose a robust self-sparse fuzzy clustering algorithm (RSSFCA) for image segmentation. The proposed RSSFCA makes two contributions. The first concerns a regularization under Gaussian metric that is integrated into the objective function of fuzzy clustering algorithms to obtain fuzzy membership with sparsity, which reduces a proportion of noisy features and improves clustering results. The second concerns a connected-component filtering based on area density balance strategy (CCF-ADB) that is proposed to address the problem of image over-segmentation. Compared to the integration of local spatial information into the objective functions, the presented CCF-ADB is simpler and faster for the removal of small areas. Experimental results show that the proposed RSSFCA addresses two problems in current fuzzy clustering algorithms, i.e., the outlier sensitivity and the over-segmentation, and it provides better image segmentation results than state-of-the-art algorithms. 10.13039/501100001809-National Natural Science Foundation of China (Grant Number: 61871259, 61811530325 (International Exchange Cooperation (IEC)\NSFC\170396, Royal Society, U.K.), 61871260, 61672333 and 61861024)

The V-Net based 3D fully convolutional neural networks have been widely used in liver volumetric data segmentation. However, due to the large number of parameters of these networks, 3D FCNs suffer from high computational cost and GPU memory usage. To address these issues, we design a lightweight V-Net (LV-Net) for liver segmentation in this paper. The proposed network makes two contributions. The first is that we design an inverted residual bottleneck block (IRB block) and a 3D average pooling block and apply them to the proposed LV-Net. Compared with vanilla convolution, depth-wise convolution and point-wise convolution employed by the IRB block can not only reduce the number of parameters significantly, but also extract features sufficiently well by decoupling cross-channel corrections and spatial correlations. The second is that the LV-Net employs 3D deep supervision to improve the final loss function in training phase, which makes the proposed LV-Net acquire a more powerful discrimination capability between liver areas and non-liver areas. The proposed LV-Net is evaluated on public LiTS dataset, and experiments demonstrate that the proposed LV-Net is superior to popular 2D and 3D networks in terms of segmentation performance, parameter quantity and computational cost.

Face recognition has made significant progress in recent years. It has been successfully applied to many applications from security entrance systems for staff identification to visual surveillance. However, the performance of current systems are highly affected by factors such as subject’s head orientation, facial expressions, and face wear-on. In order to address this challenge, researchers usually resort to more training data or more complex recognition models. We believe the secret lies in using a more rich data source. Therefore, in this paper, we use a holoscopic 3D (H3D) imaging system. To show the effectiveness of the sensor, we use it with a simple back propagation neural network (BPNN) classifier. We illustrate the accuracy of the system through an experiment and show that it has high accuracy and is robust against challenges aforementioned.

With the rapid development of deep learning (DL) in recent years, automatic modulation recognition (AMR) with DL has achieved high accuracy. However, aiming to obtain higher classification accuracy, DL requires numerous training samples. In order to solve this problem, it is a challenge to study how to efficiently use DL for AMR in the case of few samples. In this letter, inspired by the capsule network (CapsNet), we propose a new network structure named AMR-CapsNet to achieve higher classification accuracy of modulation signals with fewer samples, and further analyze the adaptability of DL models in the case of few samples. The simulation results demonstrate that when 3% of the dataset is used to train and the signal-to-noise ratio (SNR) is greater than 2 dB, the overall classification accuracy of the AMR-CapsNet is greater than 80%. Compared with convolutional neural network (CNN), the classification accuracy is improved by 20%.

The EmoPain 2020 Challenge is the first international competition aimed at creating a uniform platform for the comparison of machine learning and multimedia processing methods of automatic chronic pain assessment from human expressive behaviour, and also the identification of pain-related behaviours. The objective of the challenge is to promote research in the development of assistive technologies that help improve the quality of life for people with chronic pain via real-time monitoring and feedback to help manage their condition and remain physically active. The challenge also aims to encourage the use of the relatively underutilised, albeit vital bodily expression signals for automatic pain and pain-related emotion recognition. This paper presents a description of the challenge, competition guidelines, bench-marking dataset, and the baseline systems' architecture and performance on the three sub-tasks: pain estimation from facial expressions, pain recognition from multimodal movement, and protective movement behaviour detection., Comment: 8 pages

Finger microgestures have been widely used in human computer interaction (HCI), particularly for interactive applications, such as virtual reality (VR) and augmented reality (AR) technologies, to provide immersive experience. However, traditional 2D image-based microgesture recognition suffers from low accuracy due to the limitations of 2D imaging sensors, which have no depth information. In this article, we proposed an innovative 3D microgesture recognition system based on a holoscopic 3D imaging sensor. Due to the lack of holoscopic 3D datasets, a comprehensive holoscopic 3D microgesture (HoMG) database is created and used to develop a robust 3D microgesture recognition method. Then, a fast algorithm is proposed to extract multiviewpoint images from one holoscopic image. Furthermore, we applied a CNN model with an attention-based residual block to each viewpoint image to improve the algorithm performance. Finally, bagging classification tree decision-level fusion is applied to combine the predictions. The experimental results demonstrate that the proposed method outperforms state-of-the-art methods and delivers a better accuracy than existing methods.

A human being’s cognitive system can be simulated by artificial intelligent systems. Machines and robots equipped with cognitive capability can automatically recognize a humans mental state through their gestures and facial expressions. In this paper, an artificial intelligent system is proposed to monitor depression. It can predict the scales of Beck depression inventory II (BDI-II) from vocal and visual expressions. First, different visual features are extracted from facial expression images. Deep learning method is utilized to extract key visual features from the facial expression frames. Second, spectral low-level descriptors and mel-frequency cepstral coefficients features are extracted from short audio segments to capture the vocal expressions. Third, feature dynamic history histogram (FDHH) is proposed to capture the temporal movement on the feature space. Finally, these FDHH and audio features are fused using regression techniques for the prediction of the BDI-II scales. The proposed method has been tested on the public Audio/Visual Emotion Challenges 2014 dataset as it is tuned to be more focused on the study of depression. The results outperform all the other existing methods on the same dataset.

Immersive technology attempts to emulate a physical world through the means of a digital or simulated world. Micro-gestures are small variation actions on human hands defined by user that is one of the most convenient human action in immersive technology. Holoscopic 3D imaging uses bionics technology to capture spatial image in the pattern of “fly’s eye” and it has fruitful 3D cubic information compared to 2D images that can be used for high accurate micro-gesture controller systems. In this paper, a new micro-gesture recognition system based on holoscopic 3D imaging system is proposed for immersive applications. It is built on fast pre-processing, dynamic image feature extraction and a non-linear Support Vector Machine classifier. It is evaluated on the public Holoscopic Micro 3D Gesture (HoMG) dataset outperforming all the existing state-of-the-art methods on the same dataset.

Morphological reconstruction (MR) is often employed by seeded image segmentation algorithms such as watershed transform and power watershed as it is able to filter seeds (regional minima) to reduce over-segmentation. However, MR might mistakenly filter meaningful seeds that are required for generating accurate segmentation and it is also sensitive to the scale because a single-scale structuring element is employed. In this paper, a novel adaptive morphological reconstruction (AMR) operation is proposed that has three advantages. Firstly, AMR can adaptively filter useless seeds while preserving meaningful ones. Secondly, AMR is insensitive to the scale of structuring elements because multiscale structuring elements are employed. Finally, AMR has two attractive properties: monotonic increasingness and convergence that help seeded segmentation algorithms to achieve a hierarchical segmentation. Experiments clearly demonstrate that AMR is useful for improving algorithms of seeded image segmentation and seed-based spectral segmentation. Compared to several state-of-the-art algorithms, the proposed algorithms provide better segmentation results requiring less computing time. Source code is available at https://github.com/SUST-reynole/AMR.

Copyright © 2019 The authors. Ultrasonic Testing (UT) is one of the most important technologies in Non-Detective Testing (NDT) methods. Recently, Barker code and Golay code pairs as coded excitation signals have been applied in ultrasound imaging system with improved quality. However, the signal-to-noise ratio (SNR) of existing UT system based on Barker code or Golay code can be influenced under high high attenuation materials or noisy conditions. In this paper, we apply the convolution of Barker and Golay codes as coded excitation signals for low voltage UT devices that combines the advantages of Barker code and Golay code together. There is no need to change the hardware of UT system in this method. The proposed method has been analyzed theoretically and then in extensive simulations. The experimental results demonstrated that the main lobe level of the code produced by convolution of Barker code and Golay code pairs is much higher than the simple pulse and the main lobe of the combined code is higher than the traditional Barker code, sidelobe is the same as the baker code that constitutes this combined code. So the peak sidelobe level (PSL) of the combined code is lower than the traditional Barker code. Equipped with this, UT devices can be applied in low voltage situations.

Active thermography provides infrared images that contain sub-surface defect information, while visible images only reveal surface information. Mapping in- frared information to visible images o ers more comprehensive visualization for decision-making in rail inspection. However, the common information for regis- tration is limited due to di erent modalities in both local and global level. For example, rail track which has low temperature contrast reveals rich details in visible images, but turns blurry in the infrared counterparts. This paper pro- poses a registration algorithm called Edge-Guided Speeded-Up-Robust-Features (EG-SURF) to address this issue. Rather than sequentially integrating local and global information in matching stage which su ered from buckets e ect, this al- gorithm adaptively integrates local and global information into a descriptor to gather more common information before matching. This adaptability consists of two facets, an adaptable weighting factor between local and global information, and an adaptable main direction accuracy. The local information is extracted using SURF while the global information is represented by shape context from edges. Meanwhile, in shape context generation process, edges are weighted ac- cording to local scale and decomposed into bins using a vector decomposition manner to provide more accurate descriptor. The proposed algorithm is qual- itatively and quantitatively validated using eddy current pulsed thermography scene in the experiments. In comparison with other algorithms, better perfor- mance has been achieved. This work is supported by Engineering and Physical Sciences Research Coun- cil (EPSRC) [grant numbers EP/K503885/1]. Chaoqing Tang would also like to thank China Scholarship Council for funding his Ph.D study.

Pain -related emotions are a major barrier to effective self rehabilitation in chronic pain. Automated coaching systems capable of detecting these emotions are a potential solution. This paper lays the foundation for the development of such systems by making three contributions. First, through literature reviews, an overview of how chronic pain is expressed and the motivation for detecting it in physical rehabilitation is provided. Second, a fully labelled multimodal dataset containing high resolution multiple-view face videos, head mounted and room audio signals, full body 3-D motion capture and electromyographic signals from back muscles is supplied. Natural unconstrained pain related facial expressions and body movement behaviours were elicited from people with chronic pain carrying out physical exercises. Both instructed and non-instructed exercises where considered to reflect different rehabilitation scenarios. Two sets of labels were assigned: level of pain from facial expressions annotated by eight raters and the occurrence of six pain-related body behaviours segmented by four experts. Third, through exploratory experiments grounded in the data, the factors and challenges in the automated recognition of such expressions and behaviour are described, the paper concludes by discussing potential avenues in the context of these findings also highlighting differences for the two exercise scenarios addressed. This work was funded by the EPSRC Emotion & Pain Project (EP/H017178/1, EP/H017194/1, EP/H016988/1).

In this paper, we propose a novel approach based on a symmetric fully convolutional network within pyramid pooling (FCN-PP) for landslide mapping (LM). The proposed approach has three advantages. Firstly, this approach is automatic and insensitive to noise because multivariate morphological reconstruction (MMR) is used for image preprocessing. Secondly, it is able to take into account features from multiple convolutional layers and explore efficiently the context of images, which leads to a good tradeoff between wider receptive field and the use of context. Finally, the selected pyramid pooling module addresses the drawback of single-scale pooling employed by convolutional neural network (CNN), fully convolutional network (FCN), U-Net, etc. Experimental results show that the proposed FCN-PP is effective for LM, and it outperforms state-of-the-art approaches in terms of four metrics, Precision, Recall, F -score, and Accuracy.

Three-dimensional (3D) convolutional networks have been proven to be able to explore spatial context and spectral information simultaneously for super-resolution (SR). However, such kind of network can’t be practically designed very ‘deep’ due to the long training time and GPU memory limitations involved in 3D convolution. Instead, in this paper, spatial context and spectral information in hyperspectral images (HSIs) are explored using Two-dimensional (2D) and One-dimenional (1D) convolution, separately. Therefore, a novel 2D-1D generative adversarial network architecture (2D-1D-HSRGAN) is proposed for SR of HSIs. Specifically, the generator network consists of a spatial network and a spectral network, in which spatial network is trained with the least absolute deviations loss function to explore spatial context by 2D convolution and spectral network is trained with the spectral angle mapper (SAM) loss function to extract spectral information by 1D convolution. Experimental results over two real HSIs demonstrate that the proposed 2D-1D-HSRGAN clearly outperforms several state-of-the-art algorithms.

Active Shape Model (ASM) has been successfully applied in the segmentation of Diffusion Tensor Magnetic Resonance Image (DT-MRI, referred to as DTI) of brain. However, due to multiple anatomical structure types, irregular shapes, small gray-scale and large amount of these images, perfect segmentation performance could not be achieved. Especially, it is sensitive to initial values with high computational complexity. In this paper, we introduce the gray information of multiple atlases and the prior information of target shapes into the ASM and propose the Multi-Atlas Active Shape Model (referred to as MA-ASM) approach for DTI segmentation. It was evaluated in a manually labeled database with 7 Region of Interest (ROI)s for each of 20 subjects. In comparison with the state of art method of STAPLE (Simultaneous Truth Performance Level Estimation), the proposed algorithm was closer to the manual segmentation shape by subjective visual effects, and had higher overlap rates and lower error detection rates on quantitative analysis than STAPLE.

Because emotions are such a significant and integral part of being human, understanding them and knowing how to react to the emotions of others is a fundamental requirement for successful social interaction. We recognize emotions primarily through speech and facial expression. The topic is gaining importance in academic research triggered by research in new techniques such as identifying emotions based on speech context. This investigates the relationship between emotions and the content of our speech. This paper proposes how emotion in speech and facial expression can be recognized in real time using a framework consisting of mobile phone technology backed by cloud computing. This functionality was developed and built into a mobile phone application. Currently, the application works on any Android smartphone to detect and identify emotions in real time. The results are expressed as a percentage of all the possible emotions, such as sad, happy, fear, surprise, anger and so on. The results of the experiment confirm that face and speech emotion recognition was conducted successfully using a smartphone. It was correct in 97.26% of instances when used with standard corpora: a.Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) b.the Surrey Audio-Visual Expressed Emotion (SAVEE)

The emergence of Speech Emotion Recognition (SER) as a focal point of research into speech processing reflects its significance in the field of Human-Computer Interaction (HCI). It is core required functionality for a variety of applications and a high degree of accuracy is critical; activities as diverse as evaluating levels of emotion in children in care and measuring customer satisfaction. The extent of demand for accurate SER is reflected in the significant number of papers that have been published and studies performed. An innovative approach to speech recognition is presented in this paper centered on a cloud model alongside the conventional system for measuring emotion in speech. There are multiple stages involved in detecting and identifying emotions in speech from audio clips. The initial pre-processing stage detects the speech in the audio file and applies noise reduction. Next, the system uses Mel-frequency cepstral coefficient (MFCC) algorithms to extract features.This process results in the creation of testing and training datasets populated with the emotions: Neutral, Happiness, Sadness, Fear, Surprise, Disgust and Anger. The classification stage utilizes Support Vector Machine (SVM) classifiers to identify the emotion. An additional step implements a Confusion Matrix (CM) method to assess how these classifiers performed. Testing was executed against RAVDESS and SAVEE databases, where the detection rate achieved against the RAVDESS database was 95.3%.

In the field of machine learning, it is still a critical issue to identify and supervise the learned representation without manually intervening or intuition assistance to extract useful knowledge or serve for the downstream tasks. In this work, we focus on supervising the influential factors extracted by the variational autoencoder (VAE). The VAE is proposed to learn independent low dimension representation while facing the problem that sometimes pre-set factors are ignored. We argue that the mutual information of the input and each learned factor of the representation plays a necessary indicator of discovering the influential factors. We find the VAE objective inclines to induce mutual information sparsity in factor dimension over the data intrinsic dimension and therefore result in some non-influential factors whose function on data reconstruction could be ignored. We show mutual information also influences the lower bound of VAE’s reconstruction error and downstream classification task. To make such indicator applicable, we design an algorithm for calculating the mutual information for VAE and prove its consistency. Experimental results on MNIST, CelebA and DEAP datasets show that mutual information can help determine influential factors, of which some are interpretable and can be used to further generation and classification tasks, and help discover the variant that connects with emotion on DEAP dataset.

Automatic continuous affect recognition from multiple modality in the wild is arguably one of the most challenging research areas in affective computing. In addressing this regression problem, the advantages of the each modality, such as audio, video and text, have been frequently explored but in an isolated way. Little attention has been paid so far to quantify the relationship within these modalities. Motivated to leverage the individual advantages of each modality, this study investigates behavioral modeling of continuous affect estimation, in multimodal fusion approaches, using Linear Regression, Exponent Weighted Decision Fusion and Multi-Gene Genetic Programming. The capabilities of each fusion approach are illustrated by applying it to the formulation of affect estimation generated from multiple modality using classical Support Vector Regression. The proposed fusion methods were applied in the public Sentiment Analysis in the Wild (SEWA) multimodal dataset and the experimental results indicate that employing proper fusion can deliver a significant performance improvement for all affect estimation. The results further show that the proposed systems is competitive or outperform the other state-of-the-art approaches.

With the rapid development of augmented reality (AR) and virtual reality (VR) technology, human-computer interaction (HCI) has been greatly improved for gaming interaction of AR and VR control. The finger micro-gesture is one of the important interactive methods for HCI applications such as in the Google Soli and Microsoft Kinect projects. However, the progress in this research is slow due to the lack of high quality public available database. In this paper, holoscopic 3D camera is used to capture high quality micro-gesture images and a new unique holoscopic 3D micro-gesture (HoMG) database is produced. The principle of the holoscopic 3D camera is based on the fly viewing system to see the objects. HoMG database recorded the image sequence of 3 conventional gestures from 40 participants under different settings and conditions. For the purpose of micro-gesture recognition, HoMG has a video subset with 960 videos and a still image subset with 30635 images. Initial micro-gesture recognition on both subsets has been conducted using traditional 2D image and video features and popular classifiers and some encouraging performance has been achieved. The database will be available for the research communities and speed up the research in this area., 6 pages, 4 figures

Social touch is an important form of social interaction. In Human Robot Interaction (HRI), touch can provide additional information to other modalities, such as audio, visual. One of the application is the robot therapy that has great social significance. In this paper, an ensemble classifier based on threeway decisions is proposed to recognize touch gestures. Firstly, features are extracted from on six perspectives and four classifiers are constructed on different scales with different pre-processing methods. . Then an ensemble classifier is used to combine the four classifiers to classify the gestures. The proposed method is tested on the public Corpus of Social Touch (Cost) dataset. The experiments results not only verify the validity of our method but also show the better accuracy of our ensemble classifier.

Meaningful facial parts can convey key cues for both facial action unit detection and expression prediction. Textured 3D face scan can provide both detailed 3D geometric shape and 2D texture appearance cues of the face which are beneficial for Facial Expression Recognition (FER). However, accurate facial parts extraction as well as their fusion are challenging tasks. In this paper, a novel system for 3D FER is designed based on accurate facial parts extraction and deep feature fusion of facial parts. In particular, each textured 3D face scan is firstly represented as a 2D texture map and a depth map with one-to-one dense correspondence. Then, the facial parts of both texture map and depth map are extracted using a novel 4-stage process consists of facial landmark localization, facial rotation correction, facial resizing, facial parts bounding box extraction and post-processing procedures. Finally, deep fusion Convolutional Neural Networks (CNNs) features of all facial parts are learned from both texture maps and depth maps, respectively and nonlinear SVMs are used for expression prediction. Experiments are conducted on the BU-3DFE database, demonstrating the effectiveness of combing different facial parts, texture and depth cues and reporting the state-of-the-art results in comparison with all existing methods under the same setting., Comment: 7 pages, 3 figures

Micro-expression is a subtle and involuntary facial expression that may reveal the hidden emotion of human beings. Spotting micro-expression means to locate the moment when the microexpression happens, which is a primary step for micro-expression recognition. Previous work in microexpression expression spotting focus on spotting micro-expression from short video, and with hand-crafted features. In this paper, we present a methodology for spotting micro-expression from long videos. Specifically, a new convolutional neural network named as SMEConvNet (Spotting Micro-Expression Convolutional Network) was designed for extracting features from video clips, which is the first time that deep learning is used in micro-expression spotting. Then a feature matrix processing method was proposed for spotting the apex frame from long video, which uses a sliding window and takes the characteristics of micro-expression into account to search the apex frame. Experimental results demonstrate that the proposed method can achieve better performance than existing state-of-art methods.

Recently, real-time facial expression recognition has attracted more and more research. In this study, an automatic facial expression real-time system was built and tested. Firstly, the system and model were designed and tested on a MATLAB environment followed by a MATLAB Simulink environment that is capable of recognizing continuous facial expressions in real-time with a rate of 1 frame per second and that is implemented on a desktop PC. They have been evaluated in a public dataset, and the experimental results were promising. The dataset and labels used in this study were made from videos, which were recorded twice from five participants while watching a video. Secondly, in order to implement in real-time at a faster frame rate, the facial expression recognition system was built on the field-programmable gate array (FPGA). The camera sensor used in this work was a Digilent VmodCAM — stereo camera module. The model was built on the AtlysTM Spartan-6 FPGA development board. It can continuously perform emotional state recognition in real-time at a frame rate of 30. A graphical user interface was designed to display the participant’s video in real-time and two-dimensional predict labels of the emotion at the same time.

In this paper we present a 5G Internet Radio-Light (IoRL) architecture for supermarkets that can be readily deployed because it utilizes unlicensed visible light and millimeter wave part of the spectrum and which is used to provide shoppers with accurate location, interaction, access to Internet and Cloud based services such as high resolution video on a Tablet PC. The paper describes the supermarket use cases, the user and functional requirements and the IoRL architecture.

Automated human emotion detection is a topic of significant interest in the field of computer vision. Over the past decade, much emphasis has been on using facial expression recognition (FER) to extract emotion from facial expressions. In this paper, the proposed system presents a novel method of facial recognition based on the cloud model, in combination with the traditional facial expression system. The process of predicting emotions from facial expression images contains several stages. The first stage of this system is the pre-processing stage, which is applied by detecting the face in images and then resizing the images. The second stage involves extracting features from facial expression images using Facial Landmarks and Center of Gravity (COG) feature extraction algorithms, which generate the training and testing datasets that contain the expressions of Anger, Disgust, Fear, Happiness, Neutrality, Sadness, and Surprise. Support Vector Machine (SVM) classifiers are then used for the classification stage in order to predict the emotion. In addition, a Confusion Matrix (CM) technique is used to evaluate the performance of these classifiers. The proposed system is tested on CK+, JAFFE, and KDEF databases. However, the proposed system achieved a prediction rate of 96.3% when Facial Landmarks and the Center of Gravity (COG)+SVM method are used.

Automated facial expression recognition (AFER) is a crucial technology to and a challenging task for human computer interaction. Previous methods of AFER have incorporated different features and classification methods and use basic testing approaches. In this paper, we employ the best feature descriptor for AFER by empirically evaluating the feature descriptors named the Facial Landmarks descriptor and the Center of Gravity descriptor. We examine each feature descriptor by considering one classification method, such as the Support Vector Machine (SVM) method, with three unique facial expression recognition (FER) datasets. In addition to test accuracies, we present confusion matrices of AFER. We also analyze the effect of using these feature and image resolutions on AFER performance. Our study indicates that the Facial Landmarks descriptor is the best choice to run AFER on mobile phones. The results of our study demonstrate that the proposed facial expression recognition on a mobile phone application is successful and provides up to 96.3% recognition accuracy.

From the last decade, researches on human facial emotion recognition disclosed that computing models built on regression modelling can produce applicable performance. However, many systems need extensive computing power to be run that prevents its wide applications such as robots and smart devices. In this proposed system, a real-time automatic facial expression system was designed, implemented and tested on an embedded device such as FPGA that can be a first step for a specific facial expression recognition chip for a social robot. The system was built and simulated in MATLAB and then was built on FPGA and it can carry out real time continuously emotional state recognition at 30 fps with 47.44% accuracy. The proposed graphic user interface is able to display the participant video and two dimensional predict labels of the emotion in real time together.

In this paper, an automatic emotion detection system is built for a computer or machine to detect the emotional state from facial expressions in human computer communication. Firstly, dynamic motion features are extracted from facial expression videos and then advanced machine learning methods for classification and regression are used to predict the emotional states. The system is evaluated on two publicly available datasets, i.e. GEMEP_FERA and AVEC2013, and satisfied performances are achieved in comparison with the baseline results provided. With this emotional state detection capability, a machine can read the facial expression of its user automatically. This technique can be integrated into applications such as smart robots, interactive games and smart surveillance systems.

Music makes our life lovely because it can affect our mental states significantly with its emotional information inside. Different people might be affected differently from the same music when they listening the music in different situation and mental states. However, the common emotion information the music can be agreed even from peoples with quite different background and cultures. An automatic emotion recognition system be proposed for the music by extracting different features from the music and machine learning method learning from common knowledge on emotional state of the trained data in this paper. Firstly, two-channel audio signals are processed, and typical audio features are extracted. Then some other features used for EEG signal analysis are also extracted. Finally, these features are combined and the random forest classifier is used for the classification. The proposed method has been used on a public music dataset for test and the experimental results demonstrate its efficiency when it compare with the state-of-the-art performance in the same dataset.

Human brain behavior is very complex and it is difficult to interpret. Human emotion might come from brain activities. However, the relationship between human emotion and brain activities is far from clear. In recent years, more and more researchers are trying to discover this relationship by recording brain signals such as electroencephalogram (EEG) signals with the associated emotion information extracted from other modalities such as facial expression. In this paper, machine learning based methods are used to model this relationship in the publicly available dataset DEAP (Database for Emotional Analysis using Physiological Signals). Different features are extracted from raw EEG recordings. Then Maximum Relevance Minimum Redundancy (mRMR) was used for feature selection. These features are fed into machine learning methods to build the prediction models to extract the emotion information from EEG signals. The models are evaluated on this dataset and satisfactory results are achieved.

Facial expression has made significant progress in recent years with many commercial systems are available for real-world applications. It gains strong interest to implement a facial expression system on a portable device such as tablet and smart phone device using the camera already integrated in the devices. It is very common to see face recognition phone unlocking app in new smart phones which are proven to be hassle free way to unlock a phone. Implementation a facial expression system in a smart phone would provide fun applications that can be used to measure the mood of the user in their daily life or as a tool for their daily monitoring of the motion in phycology studies. However, traditional facial expression algorithms are normally computing extensive and can only be implemented offline at a computer. In this paper, a novel automatic system has been proposed to recognize emotions from face images on a smart phone in real-time. In our system, the camera of the smart phone is used to capture the face image, BRIEF features are extracted and k-nearest neighbor algorithm is implemented for the classification. The experimental results demonstrate that the proposed facial expression recognition on mobile phone is successful and it gives up to 89.5% recognition accuracy. The work of Hongying Meng was supported by Brunel Research Initiative & Enterprise Fund on the project entitled “Automatic Emotional State Detection and Analysis on Embedded Devices”. This research is also partially supported by the 973 project on Network Big Data Analytics funded by the Ministry of Science and Technology, China. No. 2014CB340404.

International audience; The PASCAL Visual Object Classes Challenge ran from February to March 2005. The goal of the challenge was to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). Four object classes were selected: motorbikes, bicycles, cars and people. Twelve teams entered the challenge. In this chapter we provide details of the datasets, algorithms used by the teams, evaluation criteria, and results achieved.

In this paper we present a 5G Internet Radio-Light (IoRL) architecture for homes that can be readily deployed because it utilizes unlicensed visible light and millimeter wave part of the spectrum, which does not require Mobile Network Operator (MNO) permission to deploy and which is used to provide inhabitants of houses with accurate location, interaction, access to Internet and Cloud based services such as high resolution video on a Tablet PC. The paper describes the home use cases and the IoRL architecture. EU Horizon 2020