EMOTENET: Deep neural network for facial emotion recognition using image set classification

With the growing use of robots and intelligent machines in our day-to-day as well as in-dustrial practices, it has become immensely important for machines to understand human feelings and emotions. This work addresses one of the contemporary issues in computer vi-sion i.e, Facial Emotion Recognition (FER). Many researchers have used machine learning methods to perform FER from video, image, text and audio data. Image set classification, which is more versatile than single images, has gained popularity in recent years. Although several works have incorporated image set classification with geometrical representation methods, deep learning methods have not been yet used for facial emotion recognition. We propose a novel methodology that pioneers the implementation of deep neural net-works for FER using image set classification. The proposed network is pretrained with VGGFace weights, and is built on VGG16 network with added layers as per the task’s requirement. Predictions are based on a combination of two different voting strategies including, Majority Voting and Weighted Voting. Besides reaching superior accuracy for one of the most challenging datasets for facial analysis i.e., SFEW (Static Facial Emotion in the Wild), the proposed approach achieves state-of-the-art accuracy on FER2013 (Fa-cial Emotion Recognition 2013) and FERG (Facial Emotion Recognition Group) datasets, without data augmentation, feature extraction and any additional training data. Our ex-perimental results demonstrate that the proposed image set classification provides more accurate results than the existing state-of-the-art FER approaches. In addition, an au-tonomous robot is set up and configured that can perform tasks, such as remote view, tele-operation, and 3D-mapping.