Search

Given a face image, most of previous works in gaze estimation infer the gaze via a well-trained model with supervised training. However, the distribution of test data may be very different compared to that of training data since samples might be corrupted in real-world scenarios (e.g., taking a photo in strong light). This will lead to a gap between source domain (i.e., training data) and target domain (i.e., test data). In this paper, we first introduce self-supervised learning into our method for addressing challenging situations in gaze estimation. Moreover, existing appearance-based gaze estimation methods focus on directing towards the development of powerful regressors, which mainly utilize face and eye images simultaneously or face (eye) images only. However, the problem of inter cues between face and eye features has been largely overlooked. To this end, we propose a novel Modulation-based Adaptive Network (MANet) for gaze estimation, which uses high-level knowledge to filter the distractive information and bridges the intrinsic relationship between face and eye features. Further, we combine self-supervised learning and MANet to learn to adapt to challenging cases, such as abnormal lighting conditions and poor-quality images, by minimizing a self-supervised loss and a supervised loss jointly. The experimental results on several datasets demonstrate the effectiveness of our proposed approach with a real-time speed of 900 fps on a PC with an NVIDIA Titan RTX GPU. [ABSTRACT FROM AUTHOR]

It is an explicit mode to use the clicking points by the mouse in the interactive image segmentation, while an implicit interaction mode is to use the fixation points from the eye-tracking device. Both modes can provide a series of points. Inspired by the similarity between these two interaction modes, we propose a novel human visual system (HVS) based neural network for transferring the constrained fixation point based segmentation to the clicking point based interactive segmentation. Briefly speaking, the sequence of information transmission and processing in our model is RGB image, VGG-16 backbone, LGN-like module (LGNL) and ConvLSTM block, which correspond to the pathway of stimulus transmission and processing, i.e. stimulus, retina, lateral geniculate nucleus (LGN) and visual cortex in the HVS. First, the RGB image is fed to the VGG-16 backbone to obtain the multiple-layer feature maps. Then the LGNL is adopted to effectively incorporate edge-aware features and semantic features from different layers of the VGG-16 backbone in multiple resolutions, so as to produce rich contextual features. Finally, with the guidance of the fixation density map transformed from the fixation points, the output feature maps of LGNL are utilized to generate the segmentation map via a stack of ConvLSTM blocks in a coarse-to-fine manner. Comprehensive experiments demonstrate that the proposed HVS based neural network achieves a higher segmentation performance and outperforms seven state-of-the-art methods, and prove that the transfer from constrained fixation points to clicking points is reasonable and valid. [ABSTRACT FROM AUTHOR]