Search

Spike camera is a new type of bio-inspired vision sensor, each pixel of which perceives the brightness of the scene independently, and finally outputs 3-dimensional spatiotemporal spike streams. To bridge the spike camera and traditional frame-based vision, there is some works to reconstruct spike streams into regular images. However, the low spatial resolution (400×250) of the spike camera limits the quality of the reconstructed images. Thus, it is meaningful to explore a super-resolution reconstruction for spike streams. In this paper, we propose an end-to-end network to reconstruct high-resolution images from low-resolution spike streams. To utilize more spatiotemporal features of spike streams, our network adopts a multi-level features learning mechanism, including intra-stream feature extraction by spike encoder, inter-stream dependencies extraction based on optical flow module, and joint features learning via spike-based iterative projection. Experimental results demonstrate that our network is superior to the combination of state-of-the-art intensity image reconstruction methods and super-resolution networks on simulated and real datasets.

Detecting salient objects in videos is a very challenging task. Current state-of-the-art methods are dominated by motion based deep neural networks, among which optical flow is often leveraged as motion representation. Though with robust performance, these optical flow-based video salient object detection methods face at least two problems that may hinder their generalization and application. First, computing optical flow as a pre-processing step does not support direct end-to-end learning; second, little attention has been given to the quality of visual features due to high computational cost of spatiotemporal feature encoding. In this paper we propose a novel self-sufficient feature enhancing network (SFENet) for video salient object detection, which leverages optical flow estimation as an auxiliary task while being end-to-end trainable. With a joint training scheme of both salient object detection and optical flow estimation, its multi-task architecture can be totally self-sufficient for achieving good performance without any pre-processing. Furthermore, for improving feature quality, we design four lightweight modules in spatial and temporal domains, including cross-layer fusion, multi-level warping, spatial-channel attention and boundary-aware refinement. The proposed method is evaluated through extensive experiments on five video salient object detection datasets. Experimental results show that our SFENet can be easily trained with fast convergence speed. It significantly outperforms previous methods in terms of various evaluation metrics. Moreover, with optical flow estimation and unsupervised video object segmentation as example applications, our method also yields state-of-the-art results on standard datasets.

Recently, deep learning techniques have achieved significant improvements in unsupervised video object segmentation (UVOS). However, many of existing approach cannot accurately identify the foreground objects and the background as they commonly use the coarse temporal features (e.g. , optical flow and multi-frames attention). In this paper, we present a novel model termed Flow Edge-based Motion-Attentive Network (FEM-Net), to address the unsupervised video object segmentation problem. Firstly, a motion-attentive encoder is used to jointly learn the spatial and temporal features. Then, a Flow Edge Connect (FEC) module is designed to hallucinate edges of the ambiguous or missing region in the optical flow. During the segmentation stage, the complementary temporal feature composed by the motion-attentive feature and flow edge is fed into a decoder to infer the salient foreground objects. Experimental results on two challenging public benchmarks (i.e. DAVIS-16 and FBMS) demonstrate that the proposed FEM-Net compares favorably against the state-of-the-art methods.

We present a learning-based approach for removing unwanted obstructions, such as window reflections, fence occlusions, or adherent raindrops, from a short sequence of images captured by a moving camera. Our method leverages motion differences between the background and obstructing elements to recover both layers. Specifically, we alternate between estimating dense optical flow fields of the two layers and reconstructing each layer from the flow-warped images via a deep convolutional neural network. This learning-based layer reconstruction module facilitates accommodating potential errors in the flow estimation and brittle assumptions, such as brightness consistency. We show that the proposed approach learned from synthetically generated data performs well to real images. Experimental results on numerous challenging scenarios of reflection and fence removal demonstrate the effectiveness of the proposed method., Project page: https://alex04072000.github.io/SOLD/ Code: https://github.com/alex04072000/SOLD Extension of the CVPR 2020 paper: arXiv:2004.01180

This paper presents the evaluation of the modifying Lorentzian function on the spatial-field optical flow to examine the validity in the noisy domain of motion estimation. In the routine of the motion estimation, the frame’s motion vector is estimated by the optical flow approach where the flow of the image’s frames is caught to estimate the motion vector. Nevertheless, in the noisy domain, the preciseness of the motion vector is weakened. We operated the measurement along with several non-Gaussian noises standards through several styles of the standard image frame. The determination on error vector magnitude (EVM) was taken into account to consider the preciseness of direction and length of the motion vector (MV) in comparison with various noise resistance techniques in spatial-field optical flow approach. In the achievement results, we found that this modifying Lorentzian norm function added up in the optical flow strengthen the degree of preciseness in the estimation of the spatial-field optical flow approach in the noisy domain.

In virtual reality (VR), correct and precise estimations of user’s visual fixations and head movements can enhance the quality of experience by allocating more computation resources for analysing and rendering on the areas of interest. However, there is insufficient research about understanding the visual exploration of users when modeling VR visual attention. To bridge the gap between the saliency prediction for traditional 2D content and omnidirectional content, we construct the visual attention dataset and propose the visual saliency prediction framework for panoramic videos. Around the instantaneous viewing behavior, we propose a traditional method to adapt 2D saliency models and design a CNN-based model to better predict visual saliency. In the proposed traditional model, mechanism of visual attention and viewing behaviors are considered in the computation of edge weights on graphs which are interpreted as Markov chains. The fraction of the visual attention that is diverted to each high-clarity vision (HCV) area is estimated through equilibrium distribution of this chain. We also propose the Graph-Based CNN model. The RGB channel and optical flow form the spatial-temporal units of HCVs, from which node feature vectors are extracted. Graph convolution is used to learn the mutual information between node feature vectors of HCVs and retain geometric information. Then feature vectors are aligned according to geometry structure of equirectangular format, and the feature decoder maps the aligned feature maps to the data distribution. We also construct the dynamic omnidirectional monocular (DOM) saliency dataset with 64 diverse videos evaluated by 28 people. The subjective results show that the instantaneous viewing behavior is important in the VR experience. Extensive experiments are conducted on the dataset and the results demonstrate the effectiveness of the proposed framework. The dataset will be released to facilitate the future studies related to visual saliency prediction for 360-degree contents.

Applying image-based processing methods to original videos on a framewise level breaks the temporal consistency between consecutive frames. Traditional video temporal consistency methods reconstruct an original frame containing flickers from corresponding nonflickering frames, but the inaccurate correspondence realized by optical flow restricts their practical use. In this article, we propose a temporally broad learning system (TBLS), an approach that enforces temporal consistency between frames. We establish the TBLS as a flat network comprising the input data, consisting of an original frame in an original video, a corresponding frame in the temporally inconsistent video on which the image-based technique was applied, and an output frame of the last original frame, as mapped features in feature nodes. Then, we refine extracted features by enhancing the mapped features as enhancement nodes with randomly generated weights. We then connect all extracted features to the output layer with a target weight vector. With the target weight vector, we can minimize the temporal information loss between consecutive frames and the video fidelity loss in the output videos. Finally, we remove the temporal inconsistency in the processed video and output a temporally consistent video. Besides, we propose an alternative incremental learning algorithm based on the increment of the mapped feature nodes, enhancement nodes, or input data to improve learning accuracy by a broad expansion. We demonstrate the superiority of our proposed TBLS by conducting extensive experiments.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this paper proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average F_{measure}. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR.

In this work, we investigated the feasibility of extracting continuous respiratory parameters from a single RGB camera stationed in a short-stay ward. Based on the extracted respiration parameters, we further investigated the feasibility of using respiratory features to aid in the detection of atrial fibrillation (AF). To extract respiration, we implemented two algorithms: chest optical flow (COF) and energy variance maximization (EVM). We used COF to extract respiration from the patient’s thoracic area and EVM from the patient’s facial area. Using capnography as the reference, for average breath-to-breath rate estimation (i.e., 15-second sliding windows with 50% overlap), we achieved errors within 3 breaths per minute with COF and within 3.5 breaths per minute with EVM. To detect the presence of AF in the respiratory signal, we extracted three respiratory features from the derived COF measurements. We fed these features to a logistic regression model and achieved an average AUC value of 0.64. This result showcases the potential of using camera-based respiratory parameters as predictors for AF, or as surrogate predictors when there is no sufficient facial area in the camera’s field of view for the extraction of cardiac measurements.

OBJECTIVE: Determination of end-expiration (EE) and end-inspiration (EI) time points in the respiratory cycle in free-breathing slice image acquisitions of the thorax is one key step needed for 4D image construction via dynamic magnetic resonance imaging. The purpose of this paper is to realize the automation of the labeling process. METHODS: The diaphragm is used as a surrogate for tracking respiratory motion and determining the state of breathing. Regions of interest (ROIs) containing the hemi-diaphragms are set by human interaction to compute the optical flow matrix between two adjacent 2D time slices. Subsequently, our approach examines the diaphragm speed and direction and by considering the change in the optical flow matrix, the EE or EI points are detected. RESULTS AND CONCLUSION: The labeling accuracy for the lateral aspect of the left lung and the lateral aspect of the right lung (0.63±0.71) is significantly lower (P < 0.05) than the accuracy for other positions (0.42±0.44), but the error in almost all scenarios is less than 1 time point. By comparing between automatic and manual labeling in 12 scenarios, we found out that 9 scenarios showed no significant difference (P > 0.05) between two methods. Overall, our method is found to be highly agreeable with manual labeling and greatly shortens the labeling time, requiring less than 8 minutes/study compared to 4 hours/study for manual labeling. SIGNIFICANCE: Our method achieves automatic labeling of EE and EI points without the need for use of patient internal or external markers.

Detection of abnormal crowd behavior is one of the important tasks in real-time video surveillance systems for public safety in public places such as subway, shopping malls, sport complexes and various other public gatherings. Due to high density crowded scenes, the detection of crowd behavior becomes a tedious task. Hence, crowd behavior analysis becomes a hot topic of research and requires an approach with higher rate of detection. In this work, the focus is on the crowd management and present an end-to-end model for crowd behavior analysis. A feature extraction-based model using contrast, entropy, homogeneity, and uniformity features to determine the threshold on normal and abnormal activity has been proposed in this paper. The crowd behavior analysis is measured in terms of receiver operating characteristic curve (ROC) & area under curve (AUC) for UMN dataset for the proposed model and compared with other crowd analysis methods in literature to prove its worthiness. YouTube video sequences also used for anomaly detection.

Sequence image target tracking has two requirements of accuracy and speed. Aiming at the problem of slow speed of convolutional neural network target tracking algorithm, an improved algorithm for sequence image target tracking is designed by using a fusion of Faster MDNet and optical flow method. The target algorithm designed in this paper is improved on the basis of Multi-Domain Convolutional Neural Network (MDNet). The optical flow method is used to obtain the primary selection box first, which can effectively deal with the challenges brought by scale change and in-plane rotation, and has good tracking speed and performance. The objective algorithm of this paper is used to explore the Marangoni Effect in the melting process of solid iron tailings in molten blast furnace slag. Because the main component of iron tailings is silica, and the melting point of silica is the highest, so taking silica as the research object, an intelligent analysis of Marangoni Effect in high temperature melting process of silica particles is realized. The results show that: the target algorithm achieves the real-time standard (FPS > 24), the average Distance Precision (DP) is 97.3%, the average Success Rate (SR) is 98.02%, the average Center Location Error (CLE) is 6.2684, and the average Overlap Precision (OP) coverage is 63.86%. The target algorithm has certain advantages in five evaluation indexes: distance precision, success rate, center location error, overlap precision and frames per second.

In recent years, recognition of hand gestures has received considerable attention from the research community, with an increased interest in advanced decision support systems. In applications of this type of system, recognition of hand gestures based on vision is used in the design and development of interfaces such as that of automobile users. This type of interface allows drivers to interact with the vehicle, without affecting their concentrations. This also increases driver comfort, without compromising safety. However, the video-based gesture recognition problem is not trivial and poses problems due to: (i) intra and inter-person variations in the movement of human hand gestures; (ii) variations between people in the shape and size of the human hand; and (iii) variations in lighting; and background noise. In this paper, a new recognition system is proposed to address the problem of large scale continuous gesture recognition with depth and gray-scale input videos. The proposed recognition system contains 3 main phases. Firstly continuous gesture sequences are segmented into isolated gesture using mean of velocity information calculated based on the estimation of deep optical flow. For each isolated segment a set of relevant descriptors named deep signature features is extracted in order to characterize different intensity and spatial information describing the location, the velocity and the orientation of the movement. The features constructed for both depth and gray-scale sequences are fed to linear SVM for classification. To evaluate the effectiveness of our recognition model, it was compared with other methods using two challenging datasets: KTH and Chalearn. Indeed, a first experimental study has shown that, using the KTH dataSet and the SVM classifier, results given by our method outperforms previously published methods and it reaches 95.6% in term of accuarcy and 96% in term of precision. The second experimental study, carried out on the ChaLearn LAP ConGD data set, also showed a superiority of our method from the point of view of precision by codelivery over other methods with mean Jaccard Index of 0.5011.

Temporal action detection is a challenging task in video understanding, due to the complexity of the background and rich action content impacting high-quality temporal proposals generation in untrimmed videos. Capsule networks can avoid some limitations of the invariance caused by pooling and inability from convolutional neural networks, which can better understand the temporal relations for temporal action detection. However, because of the extremely computationally expensive procedure, capsule network is difficult to be applied to the task of temporal action detection. To address this issue, this paper proposes a novel U-shaped capsule network framework with a k-Nearest Neighbor (k-NN) mechanism of 3D convolutional dynamic routing, which we named U-BlockConvCaps. Furthermore, we build a Capsules Boundary Network (CapsBoundNet) based on U-BlockConvCaps for dense temporal action proposal generation. Specifically, the first module is one 1D convolutional layer for fusing the two-stream with RGB and optical flow video features. The sampling module further processes the fused features to generate the 2D start-end action proposal feature maps. Then, the multi-scale U-Block convolutional capsule module with 3D convolutional dynamic routing is used to process the proposal feature map. Finally, the feature maps generated from the CapsBoundNet are used to predict starting, ending, action classification, and action regression score maps, which help to capture the boundary and intersection over union features. Our work innovatively improves the dynamic routing algorithm of capsule networks and extends the use of capsule networks to the temporal action detection task for the first time in the literature. The experimental results on benchmarks THUMOS14 show that the performance of CapsBoundNet is obviously beyond the state-of-the-art methods, e.g., the mAP@tIoU=0.3, 0.4, 0.5 on THUMOS14 are improved from 63.6% to 70.0%, 57.8% to 63.1%, 51.3% to 52.9%, respectively. We also got competitive results on the action detection dataset of ActivityNet1.3.

In the last few years, visual sensors are deployed almost everywhere, generating a massive amount of surveillance video data in smart cities that can be inspected intelligently to recognize anomalous events. In this work, we present an efficient and robust framework to recognize anomalies from surveillance Big Video Data (BVD) using Artificial Intelligence of Things (AIoT). Smart surveillance is an important application of AIoT and we propose a two-stream neural network in this direction. The first stream comprises instant anomaly detection that is functional over resource-constrained IoT devices, whereas second phase is a two-stream deep neural network allowing for detailed anomaly analysis, suited to be deployed as a cloud computing service. Firstly, a self-pruned fine-tuned lightweight convolutional neural network (CNN) classifies the ongoing events as normal or anomalous in an AIoT environments. Upon anomaly detection, the edge device alerts the concerned departments and transmits the anomalous frames to cloud analysis center for their detailed evaluation in the second phase. The cloud analysis center resorts to the proposed two-stream network, modeled from the integration of spatiotemporal and optical flow features through the sequential frames. Fused features flow through a bi-directional long short-term memory (BD-LSTM) layer, which classifies them into their respective anomaly classes, e.g., assault and abuse. We perform extensive experiments over benchmarks built on top of the UCF-Crime and RWF-2000 datasets to test the effectiveness of our framework. We report a 9.88% and 4.01% increase in accuracy when compared to state-of-the-art methods evaluated over the aforementioned datasets.

Example video #2forSSIMS-Flow: UAV image velocimetry workbench tool(Official repository).

In this paper, we present a fast and robust visual-inertial odometry (VIO) algorithm, which realizes the tight coupling of monocular visual odometry (VO) and low-cost inertial measurement unit (IMU). Aiming at the problem of tracking loss caused by the interference of moving objects or blurred images in the traditional VIO algorithm, a sparse optical flow method combining edge detection algorithms is proposed. In the image preprocessing, Laplace edge detection is performed on the original image first, and an area with the most texture is searched according to the sharpened image. When tracking feature points with the sparse optical flow method, the feature points within the searched area are used as the main tracking targets to reduce the impact because of some unclear part in the image on the visual front-end. The trust region dogleg method is used for non-linear optimization in the back-end of the VIO system. The effectiveness of the proposed VIO system is validated on the public data set named EuRoC MAV and compared with the advanced VIO algorithm in recent years. Experimental results show that the new VIO system proposed in this paper has good robustness and can be applied to complex scenes such as fast motion, lighting changes, lack of features, and image blur. Compared with the traditional VIO algorithm, the pose estimation speed of the proposed VIO algorithm can be improved by 10% or more.

Depth image-based rendering (DIBR) techniques play an important role in free viewpoint videos (FVVs), which have a wide range of applications including immersive entertainment, remote monitoring, education, etc. FVVs are usually synthesized by DIBR techniques in a “blind” environment (without a reference video). Thus, an effective reference-free synthesized video quality assessment (VQA) metric is vital. At present, many image quality assessment (IQA) algorithms for DIBR-synthesized images have been proposed, but limited researches have been concerned about the quality assessment of DIBR-synthesized videos. To this end, this paper proposes a novel reference-free VQA method for synthesized videos, which operates in Spatial and Temporal Domains, dubbed as STD. The design fundamental of the proposed STD metric considers the effects of two major distortions introduced by DIBR techniques on the visual quality of synthesized videos. First, considering the geometric distortion introduced by DIBR technologies can increase high-frequency contents of the synthesized frame, the influence of the geometric distortion on the visual quality of a synthesized video can be effectively evaluated by estimating high-frequency energies of each synthesized frame in spatial domain. Second, temporal inconsistency caused by DIBR techniques brings the temporal flicker distortion, which is one of the most annoying artifacts in DIBR-synthesized videos. In temporal domain, we quantify temporal inconsistency by measuring motion differences between consecutive frames. Specifically, optical flow method is first used to estimate the motion field between adjacent frames. Then, we calculate the structural similarity of adjacent optical flow fields and further adopt the structural similarity value to weight the pixel differences of adjacent optical flow fields. Experiments show that the above two features are able to well perceive the visual quality of DIBR-synthesized videos. Furthermore, since the two features are extracted from spatial and temporal domains, respectively, we integrate them using a linear weighting strategy to obtain our STD metric, which proves advantageous over two components and the competing state-of-the-art I/VQA methods. The source code is available at https://github.com/wgc-vsfm/DIBR-video-quality-assessment.

In this paper, we improve the inter coding performance of surveillance videos by simultaneously investigating the distinct characteristics of background and foreground redundancy, and introduce two novel reference frames in a complementary manner. On one hand, a block level background reference frame (BRF) is proposed to reduce the background redundancy. The proposed scheme incorporates semantic information into the compression process, and makes use of instance segmentation to facilitate the background block decision, making the generated BRF free from foreground pollution. On the other hand, in order to handle foreground redundancy, a foreground reference frame (FRF) is generated based on Surveillance Prediction Generative Adversarial Network (SP-GAN), which utilizes previous reconstructed frames, optical flow based prediction, as well as BRF to infer the foreground objects of the to-be-coded frame. We integrate the proposed scheme into HM-16.6 software and append BRF and FRF to the reference pictures list (RPS). Simulation results demonstrate considerable superiority of the proposed scheme. In particular, by adding the proposed BRF to RPS, 3% coding gains are observed compared with the state-of-the-art BRF method. When both BRF and FRF are incorporated into RPS, 5.8% gains are achieved for surveillance video coding.

Driver attention estimation is one of the key technologies for intelligent vehicles. The existing related methods only focus on the scene image or the driver's gaze or head pose. The purpose of this article is to propose a more reasonable and feasible method based on a dual-view scene with calibration-free gaze direction. According to human visual mechanisms, the low-level features, static visual saliency map, and dynamic optical flow information are extracted as input feature maps, which combine the high-level semantic descriptions and a gaze probability map transformed from the gaze direction. A multiresolution neural network is proposed to handle the calibration-free features. The proposed method is verified on a virtual reality experimental platform that collected more than 550 000 samples and obtained a more accurate ground truth. The experiments show that the proposed method is feasible and better than the state-of-the-art methods based on multiple widely used metrics. This study also provides a discussion of the effects of different landscapes, times, and weather conditions on the performance.

Laser metal deposition (LMD) as an additive manufacturing technique became increasingly important in recent years and thus the demand for component safety. This is the reason, for the need for reliable in-situ defect detection techniques. For laser beam weld seams an optical measurement technique based on an optical flow algorithm was successfully used to define the critical straining conditions that lead to hot cracking. This algorithm was adapted for bead-on-plate weld seams on LMD deposited layers of IN718 alloy while performing external strain on the specimen in an externally loaded hot cacking test facility. The resulting transversal hot cracks along the weld seam were localized via X-Ray inspection and the type of cracking confirmed by Scanning Electron Microscopy (SEM). The strain distribution was measured in the vicinity of the solidification front and correlated to the detected hot cracks. Based on the results this technique could be adopted for LMD experiments.

Weakly-supervised Temporal Action Localization (W-TAL) aims at simultaneously classifying and locating all action instances with only video-level supervision. However, current W-TAL methods have two limitations. First, they ignore the difference in video representations between an action instance and its surrounding background when generating and scoring action proposals. Second, the unique characteristics of the RGB frames and optical flow are largely ignored when fusing these two modalities. To address these problems, we present an Action Coherence Network (ACN) for W-TAL. Its core is a new coherence loss which exploits both classification predictions and video content representations to supervise action boundary regression and thus leads to more accurate action localization results. Besides, the proposed ACN explicitly takes into account the specific characteristics of RGB frames and optical flow by training two separate sub-networks, each of which is able to generate modality-specific action proposals independently. Finally, to take advantage of the complementary action proposals generated by two streams, we introduce a novel fusion module to reconcile them and obtain the final action localization results. Experiments on the THUMOS14 and ActivityNet datasets show that the proposed ACN outperforms the state-of-the-art W-TAL methods, and is even comparable to some recent fully-supervised methods. In particular, ACN achieves an mAP of 26.4% on the THUMOS14 dataset under the IoU threshold 0.5.

Semantics and motion are two cues of essence for the success in video salient object detection. Most existing deep-learning based approaches extract semantic features by the use of only one fully convolutional network with simple stacked encoders. They simulate motion patterns of video objects with two consecutive frames being simultaneously fed into a convolutional LSTM network or a weights-sharing fully convolutional network. However, such approaches have the shortcomings of producing a coarse predicted saliency map or requiring significant computational overheads. In this paper, we present a novel approach with cascaded fully convolutional networks involving motion attention (abbreviated as CFCN-MA), to achieve real-time saliency detection in videos. Our key idea is to construct twofold fully convolutional networks in order to gain a saliency map from coarse to fine. We devise an optical flow-based motion attention mechanism to improve the prediction accuracy of the initial fully convolutional networks, using the popular FlowNet2-SD model that is efficient and effective for motion pattern recognition of distinctive objects in videos. This method can obtain a fine saliency map with a refined region of interest. Moreover, we propose a means for calculating attention-guided intersection-over-union loss (shortnamed as AIoU) to supervise the CFCN-MA model in learning a saliency map with both clear edge and complete structure. Our approach is evaluated on three popular benchmark datasets, namely DAVIS, ViSal and FBMS. Experimental results demonstrate that our method outperforms many state-of-the-art techniques while meeting the real-time demand at 27 fps.

4D seismic imaging has been widely used in CO$_2$ sequestration projects to monitor the fluid flow in the volumetric subsurface region that is not sampled by wells. Ideally, real-time monitoring and near-future forecasting would provide site operators with great insights to understand the dynamics of the subsurface reservoir and assess any potential risks. However, due to obstacles such as high deployment cost, availability of acquisition equipment, exclusion zones around surface structures, only very sparse seismic imaging data can be obtained during monitoring. That leads to an unavoidable and growing knowledge gap over time. The operator needs to understand the fluid flow throughout the project lifetime and the seismic data are only available at a limited number of times. This is insufficient for understanding the reservoir behavior. To overcome those challenges, we have developed spatio-temporal neural-network-based models that can produce high-fidelity interpolated or extrapolated images effectively and efficiently. Specifically, our models are built on an autoencoder, and incorporate the long short-term memory (LSTM) structure with a new loss function regularized by optical flow. We validate the performance of our models using real 4D post-stack seismic imaging data acquired at the Sleipner CO$_2$ sequestration field. We employ two different strategies in evaluating our models. Numerically, we compare our models with different baseline approaches using classic pixel-based metrics. We also conduct a blind survey and collect a total of 20 responses from domain experts to evaluate the quality of data generated by our models. Via both numerical and expert evaluation, we conclude that our models can produce high-quality 2D/3D seismic imaging data at a reasonable cost, offering the possibility of real-time monitoring or even near-future forecasting of the CO$_2$ storage reservoir., Comment: 15 pages, 13 figures

Even though optical flow approaches based on convolutional neural networks have achieved remarkable performance with respect to both accuracy and efficiency, large displacements and motion occlusions remain challenges for most of the existing learning-based models. To address the abovementioned issues, we propose in this paper a self-attention-based multiscale feature learning optical flow computation method with occlusion feature map prediction. First, we exploit a self-attention mechanism-based multiscale feature learning module to compensate for large displacement optical flows, and the presented module is able to capture long-range dependencies from the input frames. Second, we design a simple but effective self-learning module to acquire an occlusion feature map, in which the predicted occlusion map is utilized to correct the optical flow estimation in occluded areas. Third, we explore a hybrid loss function that integrates the photometric and smoothness losses into the classical endpoint error (EPE) based loss to ensure the accuracy and robustness of the presented network. Finally, we compare the proposed method with some state-of-the-art approaches using the MPI-Sintel and KITTI test databases. The experimental results demonstrate that the proposed method achieved a competitive performance with respect to both accuracy and robustness, and it produced the best results than other methods did under large displacements and occlusions.

In the two-phase flow particle image velocimetry (PIV) experiment of an object entering water, the mask of non-computed area and the calculation of velocity field in particle images are two key stages. Due to the complexity of the edge of the object in the particle image, the mask calibration is usually performed manually and then the PIV estimation is carried out. We propose a cascaded convolutional neural network (CNN) in this paper to implement end-to-end two-phase flow fluid motion estimation. In the first stage, the image segmentation network U-Net is used to mask the non-computational area of the image and extract the liquid phase. In the second stage, we adopt the improved deep optical flow network, which is known as recurrent allpairs field transforms (RAFT) to calculate velocity field. What’s more, the corresponding datasets are generated for training model parameters. Finally, our approach is tested on synthetic and experimental images. The experimental results indicate that our approach not only reaches accurate segmentation of the calculated liquid phase region, but also achieves a high-precision velocity field calculation. Meanwhile, this cascade CNN model has high efficiency toward real-time estimation.

Action detection plays an important role in the field of video understanding and attracts considerable attention in the last decade. However, current action detection methods are mainly based on visible videos, and few of them consider scenes with low-light, where actions are difficult to be detected by existing methods, or even by human eyes. Compared with visible videos, infrared videos are more suitable for the dark environment and resistant to background clutter. In this paper, we investigate the temporal action detection problem in the dark by using infrared videos, which is, to the best of our knowledge, the first attempt in the action detection community. Our model takes the whole video as input, a Flow Estimation Network (FEN) is employed to generate the optical flow for infrared data, and it is optimized with the whole network to obtain action-related motion representations. After feature extraction, the infrared stream and flow stream are fed into a Selective Cross-stream Attention (SCA) module to narrow the performance gap between infrared and visible videos. The SCA emphasizes informative snippets and focuses on the more discriminative stream automatically. Then we adopt a snippet-level classifier to obtain action scores for all snippets and link continuous snippets into final detection results. All these modules are trained in an end-to-end manner. We collect an Infrared action Detection (InfDet) dataset obtained in the dark and conduct extensive experiments to verify the effectiveness of the proposed method. Experimental results show that our proposed method surpasses the state-of-the-art temporal action detection methods designed for visible videos, and it also achieves the best performance compared with other infrared action recognition methods on both InfAR and Infrared-Visible datasets.

Deep-learning-based video processing has yielded transformative results in recent years. However, the video analytics pipeline is energy-intensive due to high data rates and reliance on complex inference algorithms, which limits its adoption in energy-constrained applications. Motivated by the observation of high and variable spatial redundancy and temporal dynamics in video data streams, we design and evaluate an adaptive-resolution optimization framework to minimize the energy use of multi-task video analytics pipelines. Instead of heuristically tuning the input data resolution of individual tasks, our framework utilizes deep reinforcement learning to dynamically govern the input resolution and computation of the entire video analytics pipeline. By monitoring the impact of varying resolution on the quality of high-dimensional video analytics features, hence the accuracy of video analytics results, the proposed end-to-end optimization framework learns the best non-myopic policy for dynamically controlling the resolution of input video streams to globally optimize energy efficiency. Governed by reinforcement learning, optical flow is incorporated into the framework to minimize unnecessary spatio-temporal redundancy that leads to re-computation, while preserving accuracy. The proposed framework is applied to video instance segmentation which is one of the most challenging computer vision tasks, and achieves better energy efficiency than all baseline methods of similar accuracy on the YouTube-VIS dataset., Comment: IEEE Transactions on Multimedia

Tracking fails in the optimization step of conventional KLT (Kanade-Lucas-Tomasi) feature tracker mainly due to the inadequate initial condition that falls out of the convergence region, especially when a camera rotates rapidly or shakes severely. To overcome the problem, we propose a pixel-aware gyro-aided KLT feature tracker that remains accurate and robust under fast camera-ego motion conditions. In particular, we develop a pixel-aware gyro-aided feature prediction algorithm to predict the initial optical flow and obtain the patch deformation matrix of each feature point. It increases the probability of initial estimates to locate in its convergence region. Unlike the existing methods, which assume all the tracked feature pairs were constrained by the same homography prediction matrix, our prediction matrix is adjustable for each feature as it considers the pixel coordinates in the prediction process. A geometric validation based on homography and fundamental check is also adopted to remove outlier tracks. Experimental results on both public datasets and real-world sequences demonstrate that the feature tracking accuracy and robustness can be significantly improved by the proposed method. To facilitate further development, the code is publicly available at https://github.com/weibohuang0314/ pixel aware gyro aided klt feature tracker.

Video satellite can generate video image sequences with rich dynamic information, thus providing a new way for monitoring moving objects. However, to maintain high temporal resolution, video satellite images usually sacrifice their spatial resolution. Therefore, super-resolution (SR) plays a vital role in improving the quality of video satellite images. In this article, we propose a multiframe video SR neural network (MVSRnet) for video satellite image SR reconstruction. The proposed MVSRnet consists of three main subnetworks: an optical flow estimation subnetwork (OFEnet), an upscaling subnetwork (Upnet) and an attention-based residual learning subnetwork (ARLnet). The OFEnet aims to estimate low-resolution (LR) optical flow of multiple image frames. Upnet is then constructed to enhance the resolution of both input frames and the estimated LR optical flows. Motion compensation is subsequently performed according to the high-resolution (HR) optical flows. Finally, the compensated HR cube is fed to the ARLnet to generate SR results. Different from existing video satellite image SR methods, the proposed MVSRnet is a multiframe-based method with an attention mechanism, which can merge the motion information among adjacent frames and highlight the importance of extracted features. Experiments conducted on Jilin-1 and OVS-1 video satellite images demonstrate that the proposed MVSRnet significantly outperforms some state-of-the-art SR methods.

Talking face generation is a demanding task to synthesize a high quality video with accurate lip synchronization and rhythmic head motion. Any subtle artifacts could be sensitively captured by humans and lead to poor visual quality. Existing methods tend to employ a conditional generation solution, which introduces facial landmarks to bridge the input information and output videos. However, these methods always suffer from unrealistic facial animations, because 1) they only take single-mode input, but ignore the complementarity of multimodal inputs for lip-sync improvement; 2) they only explore lip movements, but ignore the articulator synergy between lips and jaw; 3) they generate each video frame in a temporal-independent way, but ignore the temporal continuity among the entire video. To address these limitations, in this paper, we present a novel method to generate realistic and temporally coherent talking heads by considering multimodal inputs, articulator synergy, inter-frame consistency and intra-frame consistency. Firstly, for landmark prediction, a novel Multiple Synergy Network (MSN) is proposed to improve the accuracy of landmark prediction by incorporating multimodal inputs (i.e., audio and text inputs). Besides, instead of merely considering lip landmarks, we also explore the jaw movements to ensure articulator synergy among lips and jaw. Secondly, for realistic video generation, a Video Consistency Network (VCN) is proposed conditioned on the predicted landmarks. In VCN, the optical flow is adopted to model the temporal continuity between frames to ensure inter-frame consistency. Meanwhile, a mouth generation branch is proposed to enhance mouth texture and the corresponding mouth mask is employed to ensure intra-frame consistency between the mouth area and the others. Extensive experiments demonstrate that our approach exhibits excellent superiority on lip-sync and can generate photo-realistic facial animations. Project is available at http://imcc.ustc.edu.cn/project/tfgen/.

Ground-based sky imagers have become a valuable tool for cloud monitoring as they allow to observe the local sky in great detail due to their temporally and spatially high-resolution imagery. Set up in pairs, they can be used to estimate both cloud geometry and motion. While considerable effort is currently devoted to their use for cloud shadow forecasting and short-term prediction of solar irradiance for renewable energies, their potential for vertical wind profiling and cloud evolution monitoring has not yet received much attention. We evaluate the accuracy and reliability of such retrievals from two sky imagers in stereo mode by comparing derived 3D cloud motion vector fields with observations from a wind lidar. We find that the wind vectors derived from the detected motion of cumulus clouds and multiple cloud layers can be retrieved with a quality comparable to a wind lidar. Differences in wind speed and direction from both methods remain mostly below 1 to 3 ms−1 in wind speed and 6° to 20° in wind direction, depending on cloud type.

Recognizing various human actions in video is a challenging task, which is also one of the key tasks in computer vision. It has received extensive attention from AI researchers Bakker et al. (2003), Bruderlin and Williams (1995), Cardle et al. (2003), Carlsson (1996, 1999), Clausen and Kurth (2004). It has important applications in human behavior analysis, artificial intelligence, and video surveillance. Compared with still image classification, the time component of video provides important clues for recognition, so multiple human actions can be recognized based on motion information. In addition, video provides natural data enhancement for individual images. For motion recognition from videos, appearance and temporal dynamics are two key and complementary cues. In this work, we formulate a human motion recognition framework based on 2D spatial feature fusion of Kinect bone data. This method utilizes the human body structure and spatial geometry to represent human body structure and extract features. Meanwhile, it combines the active action and auxiliary action features of two spatial dimensions by layering, and leverages the pervasively used support vector machine and hidden Markov model to classify human body structure. Noticeably, it is difficult to extract its information due to the limitations of background clutter, viewpoint change, scale change, different lighting conditions and camera movement. In this way, while learning the classification information of human behavioral categories, designing effective representations is the key to deal with these challenges. In this work, a method for human motion recognition in video based on ResNext network is proposed. Based on our proposed ResNeXt network, using the data of RGB and optical flow, we can extract more appearance characteristics and temporal features of human actions, so as to better achieve the classification of actions. The utilization of video time segmentation method can capture the long range of time in video, in order to make better use of the longer range of time information in video. Comprehensive experimental results have shown that the proposed method improves the performance of UCF101 and HMDB51 motion recognition data sets to a certain extent.

Aiming to deal with underwater localization for small-size robots in GPS-denied and structured environment, this paper proposed a novel multi-sensor fusion-based self-localization system using low-cost sensors. Based on multi-sensor information fusion, an Extended Kalman Filter (EKF) is utilized to synthesize the multi-source information from an Inertial Measurement Unit (IMU), optical flow, pressure sensor and ArUco markers, which enables the robot obtain a highly precise positioning. This method also can reduce the location drift over time owing to the loss of markers in pure markers-based localization. Specially, a velocity correction model is proposed using the angle information obtained by IMU, which can compensate optical flow-based velocity estimation errors caused by robot posture changes. Finally, to validate the performance of the proposed self-localization system, simulations are conducted using Gazebo simulator on the robot operating system (ROS). Moreover, a series of experiments in an indoor swimming pool are presented. Results of the proposed method and dead reckoning are compared in simulation and experiment to demonstrate the robustness and feasibility of the proposed localization system.

Shot boundary detection is the process of segmenting a video into basic units known as shots by discovering transition frames between shots. Researches have been conducted to accurately detect the shot boundaries. However, the acceleration of the shot detection process with higher accuracy needs improvement. A new method was introduced in this paper to find out the boundaries of abrupt shots in the video with high accuracy and lower computational cost. The proposed method consists of two stages. First, projection features were used to distinguish non boundary transitions and candidate transitions that may contain abrupt boundary. Only candidate transitions were conserved for next stage. Thus, the speed of shot detection was improved by reducing the detection scope. In the second stage, the candidate segments were refined using motion feature derived from the optical flow to remove non boundary frames. The results manifest that the proposed method achieved excellent detection accuracy (0.98 according to F-Score) and effectively speeded up detection process. In addition, the comparative analysis results confirmed the superior performance of the proposed method versus other methods.

This paper analyzes the simulation of special effects animation through fluid equations and data-driven methods. This paper also considers the needs of computer fluid animation simulation in terms of computational accuracy and simulation efficiency, takes high real-time, high interactivity, and high physical accuracy of simulation algorithm as the research focus and target, and proposes a solution algorithm and acceleration scheme based on deep neural network framework for the key problems of simulation of natural phenomena including smoke and liquid. With the deep development of artificial intelligence technology, deep neural network models are widely used in research fields such as computer image classification, speech recognition, and fluid detail synthesis with their powerful data learning capability. Its stable and efficient computational model provides a new problem-solving approach for computerized fluid animation simulation. In terms of time series reconstruction, this paper adopts a tracking-based reconstruction method, including target tracking, 2D trajectory fitting and repair, and 3D trajectory reconstruction. For continuous image sequences, a linear dynamic model algorithm based on pyramidal optical flow is used to track the feature centers of the objects, and the spatial coordinates and motion parameters of the feature points are obtained by reconstructing the motion trajectories. The experimental results show that in terms of spatial reconstruction, the matching method proposed in this paper is more accurate compared with the traditional stereo matching algorithm; in terms of time series reconstruction, the error of target tracking reduced. Finally, the 3D motion trajectory of the point feature object and the motion pattern at a certain moment are shown, and the method in this paper obtains more ideal results, which proves the effectiveness of the method.

The neurological disorders are developed in adults due to reduced visual perception. Opto Kinetic Nystagmus (OKN) is a clinical method to detect visual perception. For objective measurements, a computational algorithm based OKN detection is preferable rather than clinical practice. In this paper, a memory-efficient Subsampled Lucas-Kanade Optical Flow (SLKOF) is proposed. The proposal employs the Subsampling of images for various levels. The proposal deals with the computation of OKN gain for different image Subsampling factors using the MATLAB platform. The experimental set up to observe OKN is done using computer-based rotation control of the drum through a stepper motor. The results are compared with the well established Lucas-Kanade (LK) method for Optical flow. It is observed that OKN gain corresponds to 1/4th of a subsampled image of the SLKOF method correlates with the LK method for the majority of the cases. This validation evidently elucidates that the proposal is computationally efficient.

Action recognition is a fundamental and challenging task in computer vision. In recent years, optical flow, as the auxiliary information of frames in a video, has been widely applied to action recognition because of its advantage of utilizing the motion information of video data. However, existing methods only fuse the score of classification probabilities of the two streams; they do not consider the interaction between the image frames and the optical flows. In addition, the other important challenges lie in capturing significant motion information to be able to recognize the action. To overcome these problems, an action recognition model based on a multi-view temporal attention mechanism is proposed in this paper. Specifically, global temporal attention pooling is first designed to fuse multiple frame image features, where more attention is given to discriminative frames. Second, considering the complementarity of the image frame and optical flow, feature-level multi-view fusion methods are proposed. Experiments on three widely used benchmark datasets on action recognition show that our method outperforms other existing state-of-the-art methods. In addition, the effectiveness of the proposed method is extensively demonstrated under different factors, such as the temporal attention pooling strategy, multi-view feature fusion and network architecture. The promising experimental results demonstrate that introducing the temporal attention layer and feature-level multi-view fusion methods is of great effectiveness and overcomes the shortcomings of classical two-stream networks to some extent. Specifically, the proposed method has the following advantages. First, the temporal attention layer can accurately capture key frames that are more conducive to recognizing actions. Second, two kinds of features from image frames and optical flows are combined to make full use of their complementarity. Finally, a variety of fusion methods are employed for feature-level fusion instead of straightforward score fusion.

Made available in DSpace on 2022-05-01T10:18:56Z (GMT). No. of bitstreams: 0 Previous issue date: 2022-02-01 This study examined changes in postural control and gaze performance of faller and non-faller older adults under conditions of visual tasks and optical flow manipulations. Fifteen older non-fallers (69.8 years, ± 3.2) and fifteen older fallers (71.1 years, ± 6.4) stood on a force platform inside a moving room wearing an eye tracker. Four tasks were performed: gaze fixation; predictable saccades; unpredictable saccades; and free-viewing. The stimuli appeared at a frequency of 1.1 Hz during conditions of predictable and unpredictable saccades. Sixteen trials were divided into two blocks. In the first block, the room remained stationary. In the second block, the room oscillated, without the participant’s awareness, with a 0.6 cm amplitude and 0.2 Hz frequency. Results showed postural sway attenuation in older fallers during the saccadic tasks compared to gaze fixation and free-viewing tasks, in both stationary and moving room conditions. Both groups showed increased center of pressure (CoP) magnitude during the moving room condition and CoP displacements strongly coupled to the room’s movement. The influence of the moving room on the postural sway was reduced during the saccadic tasks for both older groups. Older fallers exhibited higher variability compared to older non-fallers. Gaze behavior differences between groups were dependent on the goals of the visual tasks. Therefore, CoP displacements of older adults are reduced during saccadic tasks regardless of their falling history. However, postural and gaze performance of older fallers suggests increased deterioration of postural and oculomotor control which may be used as a predictor of fall risk. Faculty of Sciences Department of Physical Education Laboratory of Information Vision and Action (LIVIA) São Paulo State University (UNESP), Av. Eng. Luiz Edmundo Carrijo Coube, 14-01, Vargem Limpa, São Paulo Graduate Program in Movement Sciences São Paulo State University - UNESP, São Paulo Faculty of Sciences Department of Physical Education Human Movement Research Laboratory (MOVI-LAB) São Paulo State University (UNESP), São Paulo Institute of Bioscience Department of Physical Education Movement Studies Laboratory (LEM) São Paulo State University (UNESP), São Paulo Faculty of Sciences Department of Physical Education Laboratory of Information Vision and Action (LIVIA) São Paulo State University (UNESP), Av. Eng. Luiz Edmundo Carrijo Coube, 14-01, Vargem Limpa, São Paulo Graduate Program in Movement Sciences São Paulo State University - UNESP, São Paulo Faculty of Sciences Department of Physical Education Human Movement Research Laboratory (MOVI-LAB) São Paulo State University (UNESP), São Paulo Institute of Bioscience Department of Physical Education Movement Studies Laboratory (LEM) São Paulo State University (UNESP), São Paulo

Scene flow estimation is one of the most crucial components of many scene understanding tasks, which represents the complete 3D motion of objects in dynamic scene. Most of existing scene flow estimation approaches are usually based on joint learning framework, which cast the problem of scene flow estimation to dense prediction of optical flow and stereo matching. However, these approaches need to reconstruct 3D motion from optical flow and disparity using 2D stereo image pairs, which leads the estimation process to be indirect. Recently, FlowNet3D attempts to learn scene flow from 3D point clouds, which adopts element-wise max-pooling to aggregate features from different points. Nevertheless, max-pooling operation only can obtain the strongest activation on features across a local or global region, which may increase the loss of some useful detailed and contextual. Specifically, for dense estimation task, the ability to transfer information gradually from coarse to fine layers is important. To address this problem, we investigate a new deep architecture, 3D point convolutional network, to learn scene flow from 3D point clouds. This specific architecture uses multi-layer perceptron (MLP) to approximate the weight function and applies a density scale to re-weight the learned weight functions for each convolutional filter, which can make the network permutation-invariant and translation invariant on 3D space which is beneficial for feature aggregation. Extensive experimental results are conducted on FlyingThings3D and KITTI scene flow datasets and show the effectiveness of our proposed approach. Our algorithm can achieve state-of-the-art performance on FlyingThings3D and KITTI datasets.

The turbulent energy dissipation rate is an important parameter that determines the transfer rates of mass, heat and momentum in chemical engineering industry. Especially in a multiphase reactor, the local instantaneous turbulent dissipation rate affects the breakage and coalescence of bubbles or droplets. To directly determine the local turbulent dissipation rate, fluctuating velocity gradients have to be measured down to the Kolmogorov length scale. This paper compares the advantages and disadvantages of three different methods, namely, correlation method, optical flow method and hybrid method, in the evaluation of velocity fields, which are subsequently used to calculate the turbulent dissipation rate. An instantaneous flow field of homogeneous isotropic turbulence from DNS results is used as a benchmark. The velocity field and turbulent dissipation rate field obtained by these three methods are compared to the DNS results. It is shown that the hybrid method performs better in both velocity field evaluation and local turbulent dissipation rate estimation compared to the other two methods. This is because the hybrid method combines the advantages of the correlation method in achieving a stable averaged velocity field and the optical flow method in achieving pixel-level resolution measurement of the flow field.

Ghosting artifacts caused by moving objects and misalignments are a key challenge in constructing high dynamic range (HDR) images. Current methods first register the input low dynamic range (LDR) images using optical flow before merging them. This process is error-prone, and often causes ghosting in the resulting merged image. We propose a novel dual-attention-guided end-to-end deep neural network, called DAHDRNet, which produces high-quality ghost-free HDR images. Unlike previous methods that directly stack the LDR images or features for merging, we use dual-attention modules to guide the merging according to the reference image. DAHDRNet thus exploits both spatial attention and feature channel attention to achieve ghost-free merging. The spatial attention modules automatically suppress undesired components caused by misalignments and saturation, and enhance the fine details in the non-reference images. The channel attention modules adaptively rescale channel-wise features by considering the inter-dependencies between channels. The dual-attention approach is applied recurrently to further improve feature representation, and thus alignment. A dilated residual dense block is devised to make full use of the hierarchical features and increase the receptive field when hallucinating missing details. We employ a hybrid loss function, which consists of a perceptual loss, a total variation loss, and a content loss to recover photo-realistic images. Although DAHDRNet is not flow-based, it can be applied to flow-based registration to reduce artifacts caused by optical-flow estimation errors. Experiments on different datasets show that the proposed DAHDRNet achieves state-of-the-art quantitative and qualitative results.

Stereo vision is widely deployed on robots and drones to enable depth estimation at a low cost. The combination of lightweight deep neural network (DNN) and cost volumes algorithm is proved to possess the advantages of both high depth estimation accuracy and speed. However, currently there is no accelerator architecture compatible with both efficient DNN inference and cost generation algorithms such as stereo matching. This work proposes a stereo vision accelerator called Dadu-eye, dedicated to real-time processing of high-resolution image streams. The proposed architecture adopts a pipelined hardware design with the techniques of operation approximation and scheduling-level optimization. First, a cost estimation block is designed to generate cost volumes from both luminance and color information. Second, a super pipelined multiplication and accumulation array with a row scan-based fused-layer convolution scheduling is proposed to perform the encoding and decoding neural network efficiently. Finally, an optical flow block is designed and cooperates with the array to approximately predict half of the frames’ depth to achieve real-time (30fps) processing on 1080p view. Based on the SMIC 40 nm CMOS process, this stereo vision accelerator achieves 5.3 TOPS/W power efficiency and significantly reduces 81% off-chip memory access.

Face recognition systems are widely used for target recognition and identity authentication, such as automated teller machines, mobile phones, and entrance guard systems. However, face recognition systems are vulnerable to presentation attacks, such as photo, replay, and 3D mask attacks. In particular, silicone mask attacks pose a greater threat to face recognition systems because high-quality silicone masks do living properties. To promote the development of face anti-spoofing detection algorithms for silicone mask attacks, this paper constructs a Silicone Mask Face Motion Video Dataset (SMFMVD) containing 200 real face videos and 200 silicone mask face videos. These videos include different facial motions collected from 20 subjects. Moreover, inspired by the observation that the silicone mask face’s facial movement is not so natural as the real face, we propose a novel silicone mask face anti-spoofing detection method based on visual saliency and facial motion characteristics. Specifically, we compute the visual saliency map of a given face image by simulating two kinds of eye movement behaviors, namely “gaze” and “saccade”. Then, we propose a saliency-weighted histogram of local binary pattern operator to extract facial texture features in spatial domain and a saliency-guided histogram of oriented optical flow operator to extract facial motion features in temporal domain. Finally, the support vector machine is used to fuse two groups of facial features to distinguish real and spoof faces. Extensive experiments on public and self-built datasets show its superiority over the state-of-the-art methods.

Motion estimation is the basic need for the success of many video analysis algorithms such as moving object detection, human activity recognition, etc. Most of the motion estimation algorithms are prone to weather conditions and thus, they fail to estimate the motion in degraded weather. Severe weather situations like snow, rain, haze, smog, etc., degrades the performance and reliability of video analysis algorithms. In this paper, we have analyzed the effect of the haze on motion estimation in hazy videos. We propose a cascaded architecture i.e. haze removal followed by optical flow for motion estimation in hazy videos. The proposed image de-hazing network is build upon the Residual and Inception module concepts and named as ResINet. Further, an optical flow is utilized to estimate the motion information. We have carried out the visual analysis to validate the proposed approach for motion estimation in hazy videos. Also, to validate the proposed ResINet for de-hazing, we carried out the quantitative analysis on two benchmark image de-hazing datasets.