Search

Applying image-based processing methods to original videos on a framewise level breaks the temporal consistency between consecutive frames. Traditional video temporal consistency methods reconstruct an original frame containing flickers from corresponding nonflickering frames, but the inaccurate correspondence realized by optical flow restricts their practical use. In this article, we propose a temporally broad learning system (TBLS), an approach that enforces temporal consistency between frames. We establish the TBLS as a flat network comprising the input data, consisting of an original frame in an original video, a corresponding frame in the temporally inconsistent video on which the image-based technique was applied, and an output frame of the last original frame, as mapped features in feature nodes. Then, we refine extracted features by enhancing the mapped features as enhancement nodes with randomly generated weights. We then connect all extracted features to the output layer with a target weight vector. With the target weight vector, we can minimize the temporal information loss between consecutive frames and the video fidelity loss in the output videos. Finally, we remove the temporal inconsistency in the processed video and output a temporally consistent video. Besides, we propose an alternative incremental learning algorithm based on the increment of the mapped feature nodes, enhancement nodes, or input data to improve learning accuracy by a broad expansion. We demonstrate the superiority of our proposed TBLS by conducting extensive experiments.

In recent years, recognition of hand gestures has received considerable attention from the research community, with an increased interest in advanced decision support systems. In applications of this type of system, recognition of hand gestures based on vision is used in the design and development of interfaces such as that of automobile users. This type of interface allows drivers to interact with the vehicle, without affecting their concentrations. This also increases driver comfort, without compromising safety. However, the video-based gesture recognition problem is not trivial and poses problems due to: (i) intra and inter-person variations in the movement of human hand gestures; (ii) variations between people in the shape and size of the human hand; and (iii) variations in lighting; and background noise. In this paper, a new recognition system is proposed to address the problem of large scale continuous gesture recognition with depth and gray-scale input videos. The proposed recognition system contains 3 main phases. Firstly continuous gesture sequences are segmented into isolated gesture using mean of velocity information calculated based on the estimation of deep optical flow. For each isolated segment a set of relevant descriptors named deep signature features is extracted in order to characterize different intensity and spatial information describing the location, the velocity and the orientation of the movement. The features constructed for both depth and gray-scale sequences are fed to linear SVM for classification. To evaluate the effectiveness of our recognition model, it was compared with other methods using two challenging datasets: KTH and Chalearn. Indeed, a first experimental study has shown that, using the KTH dataSet and the SVM classifier, results given by our method outperforms previously published methods and it reaches 95.6% in term of accuarcy and 96% in term of precision. The second experimental study, carried out on the ChaLearn LAP ConGD data set, also showed a superiority of our method from the point of view of precision by codelivery over other methods with mean Jaccard Index of 0.5011.

International audience; Representing dynamic textures (DTs) plays an important role in many real implementations in the computer vision community. Due to the turbulent and non-directional motions of DTs along with the negative impacts of different factors (e.g., environmental changes, noise, illumination, etc.), efficiently analyzing DTs has raised considerable challenges for the state-of-the-art approaches. For 20 years, many different techniques have been introduced to handle the above well-known issues for enhancing the performance. Those methods have shown valuable contributions, but the problems have been incompletely dealt with, particularly recognizing DTs on large-scale datasets. In this article, we present a comprehensive taxonomy of DT representation in order to purposefully give a thorough overview of the existing methods along with overall evaluations of their obtained performances. Accordingly, we arrange the methods into six canonical categories. Each of them is then taken in a brief presentation of its principal methodology stream and various related variants. The effectiveness levels of the state-of-the-art methods are then investigated and thoroughly discussed with respect to quantitative and qualitative evaluations in classifying DTs on benchmark datasets. Finally, we point out several potential applications and the remaining challenges that should be addressed in further directions. In comparison with two existing shallow DT surveys (i.e., the first one is out of date as it was made in 2005, while the newer one (published in 2016) is an inadequate overview), we believe that our proposed comprehensive taxonomy not only provides a better view of DT representation for the target readers but also stimulates future research activities.

Face recognition systems are widely used for target recognition and identity authentication, such as automated teller machines, mobile phones, and entrance guard systems. However, face recognition systems are vulnerable to presentation attacks, such as photo, replay, and 3D mask attacks. In particular, silicone mask attacks pose a greater threat to face recognition systems because high-quality silicone masks do living properties. To promote the development of face anti-spoofing detection algorithms for silicone mask attacks, this paper constructs a Silicone Mask Face Motion Video Dataset (SMFMVD) containing 200 real face videos and 200 silicone mask face videos. These videos include different facial motions collected from 20 subjects. Moreover, inspired by the observation that the silicone mask face’s facial movement is not so natural as the real face, we propose a novel silicone mask face anti-spoofing detection method based on visual saliency and facial motion characteristics. Specifically, we compute the visual saliency map of a given face image by simulating two kinds of eye movement behaviors, namely “gaze” and “saccade”. Then, we propose a saliency-weighted histogram of local binary pattern operator to extract facial texture features in spatial domain and a saliency-guided histogram of oriented optical flow operator to extract facial motion features in temporal domain. Finally, the support vector machine is used to fuse two groups of facial features to distinguish real and spoof faces. Extensive experiments on public and self-built datasets show its superiority over the state-of-the-art methods.

We present Supervision by Registration and Triangulation (SRT), an unsupervised approach that utilizes unlabeled multi-view video to improve the accuracy and precision of landmark detectors. Being able to utilize unlabeled data enables our detectors to learn from massive amounts of unlabeled data freely available and not be limited by the quality and quantity of manual human annotations. To utilize unlabeled data, there are two key observations: (1) the detections of the same landmark in adjacent frames should be coherent with registration, i.e., optical flow. (2) the detections of the same landmark in multiple synchronized and geometrically calibrated views should correspond to a single 3D point, i.e., multi-view consistency. Registration and multi-view consistency are sources of supervision that do not require manual labeling, thus it can be leveraged to augment existing training data during detector training. End-to-end training is made possible by differentiable registration and 3D triangulation modules. Experiments with 11 datasets and a newly proposed metric to measure precision demonstrate accuracy and precision improvements in landmark detection on both images and video. Code is available at https://github.com/D-X-Y/landmark-detection., Comment: Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2020

Traditional video compression approaches build upon the hybrid coding framework with motion-compensated prediction and residual transform coding. In this paper, we propose the first end-to-end deep video compression framework to take advantage of both the classical compression architecture and the powerful non-linear representation ability of neural networks. Our framework employs pixel-wise motion information, which is learned from an optical flow network and further compressed by an auto-encoder network to save bits. The other compression components are also implemented by the well-designed networks for high efficiency. All the modules are jointly optimized by using the rate-distortion trade-off and can collaborate with each other. More importantly, the proposed deep video compression framework is very flexible and can be easily extended by using lightweight or advanced networks for higher speed or better efficiency. We also propose to introduce the adaptive quantization layer to reduce the number of parameters for variable bitrate coding. Comprehensive experimental results demonstrate the effectiveness of the proposed framework on the benchmark datasets.

Accurately recognizing various human actions is a key technique in many AI applications, such as visual tracking and human–computer interaction. Aiming at solving the difficulty that the local spatial–temporal features at low-layer is limited and the descriptiveness of middle-level features is weak, we propose a novel human action understanding framework by leveraging the spatial–temporal depth features. More specifically, based on the fact that violent action regions encode highly discriminative information during human action recognition, we employ human depth clue of video images to identify the salient regions for each human. The optical flow characteristics in the regions are utilized as the energy function to measure the regional representativeness. The deep learning architecture is proposed to identify the regions of motions by the energy function. In this way, the sample points are distributed in the areas with intense movements. The collected sample points are utilized as the learned features to capture human action. Based on the learned deep feature, an SVM classifier is used to identify different human action. Comprehensive experimental results have shown that the average recognition accuracy of our human action recognition algorithm reaches 92%, and also exhibits a high robustness to complicated backgrounds.

Today, artificial intelligence is everywhere in people’s daily lives. Visual tracking, which is used to identify and continuously track specific targets, is an important research domain in the study of artificial intelligence. However, current visual tracking methods are not accurate enough for object tracking with background clutter, which can easily lead to tracking failures. Therefore, in this paper, in order to solve the problem of tracking failure in clutter background, we propose a template update mechanism to improve the accuracy of visual tracking. First, an original template is saved when the background clutter is detected. During background clutter, we use both the original template and the current template at the location estimated by the optical flow and choose better one. Next, the original template is reused after the background clutter is ended. Finally, the proposed mechanism is used both in the KCF and BACF algorithm to verify the effectiveness of the mechanism. With experiments on the OTB2015 dataset, results show that the proposed mechanism has improved accuracy and success rate of the two baseline algorithms. Meanwhile, in state-of-the-art algorithms, the algorithm using the proposed mechanism also has excellent tracking performance. In addition, this method also has strong tracking robustness and adaptation capability to sequential learning for video data.

Recently, deep convolutional neural networks have demonstrated great effectiveness in action recognition with both RGB and optical flow in the past decade. However, existing studies generally treat all frames and pixels equally, potentially leading to poor robustness of models. In this paper, we propose a novel parameter-free spatial–temporal pooling block (referred to as STP) for action recognition in videos to address this challenge. STP is proposed to learn spatial and temporal weights, which are further used to guide information compression. Different from other temporal pooling layers, STP is more efficient as it discards the non-informative frames in a certain clip. In addition, STP applies a novel loss function that forces the model to learn information from sparse and discriminative frames. Moreover, we introduce a dataset for ferry action classification, named Ferryboat-4, which includes four categories: Inshore, Offshore, Traffic, and Negative. This designed dataset can be used for the identification of ferries with abnormal behaviors, providing the essential information to support the supervision, management, and monitoring of ships. All the videos are acquired via real-world cameras. We perform extensive experiments on publicly available datasets as well as Ferryboat-4 and find that the proposed method outperforms several state-of-the-art methods in action classification. Source code and datasets are available at https://github.com/jiaming-wang/STP .

Precipitation nowcasting is an important research topic in meteorology, which relates to many aspects of people’s life and social development. Under the combined influence of resolution and corresponding timestamp, a certain nonlinear relationship is satisfied between the echo intensity and precipitation. Therefore, the short-term precipitation prediction scheme based on radar echo extrapolation has become the main research method. By analyzing the spatiotemporal characteristics of the radar echo images, we found that precipitation results are related not only to currently observed radar echo images but also to some non-image features such as wind speed and shape of cloud clusters. Inspired by optical flow method, combined with the characteristics of radar reflectivity, we propose a new method SCENT to achieve precipitation prediction. Firstly, the sparse correspondence method based on Fast feature detection and SIFT matching is used to radar echo extrapolate and complete the extraction of non-image influence features. Afterwards, an improved neural network is utilized for regression calculation to obtain the total precipitation. By comparing with existing prediction models based on deep neural network, our new method can make precipitation nowcasting more accurate.

License plate detection and recognition (LPDR) has attracted considerable attention in recent years, and many algorithms have presented the competitive performance on several datasets. However, there are still three significant issues to be addressed in this field. Firstly, most methods have poor detection performance in unconstrained scenarios with moving vehicles and highly distracting background objects. Secondly, existing systems generally focus on single image-based algorithms, yet traffic video sequences provide more effective information than individual frames for LPDR tasks. Thirdly, images and videos captured in complex environments may be adversely affected by distortions and low resolution, causing sensitive recognition performance and reduced robustness. To remedy these issues, we propose to automatically perform license plate detection, tracking, and recognition in real-world traffic videos and integrate them into a unified end-to-end framework via deep learning. The contributions of this paper are threefold: 1) A deep flow-guided spatiotemporal license plate detector is proposed to model the video contextual information by introducing optical flow and a novel spatiotemporal attention mechanism; 2) An online license plate tracker is developed to bridge video-based detection and recognition which utilizes both motion and deep appearance information, and innovatively, it can be end-to-end trained with the detector via multi-task learning; 3) The efficient quality-guided license plate recommender and recognizer are proposed to jointly perform stream recognition. The former recommends high-quality frames from video streams while the latter generates recognition results. We evaluate the proposed method on three traffic video-based license plate datasets, and ablation studies have been presented to verify the effectiveness of each component mentioned above. Moreover, extensive experiments are conducted for comparison with other approaches in different scenarios, and the results have demonstrated that our method achieves state-of-the-art performance on all datasets.

In this paper, a new direct computational approach to dense 3D reconstruction in autonomous driving is proposed to simultaneously estimate the depth and the camera motion for the motion stereo problem. A traditional Structure from Motion framework is utilized to establish geometric constrains for our variational model. The architecture is mainly composed of the texture constancy constraint, one-order motion smoothness constraint, a second-order depth regularize constraint and a soft constraint. The texture constancy constraint can improve the robustness against illumination changes. One-order motion smoothness constraint can reduce the noise in estimation of dense correspondence. The depth regularize constraint is used to handle inherent ambiguities and guarantee a smooth or piecewise smooth surface, and the soft constraint can provide a dense correspondence as initial estimation of the camera matrix to improve the robustness future. Compared to the traditional dense Structure from Motion approaches and popular stereo approaches, our monocular depth estimation results are more accurate and more robust. Even in contrast to the popular depth from single image networks, our variational approach still has good performance in estimation of monocular depth and camera motion.

Pedestrian detection has always been a research hotspot in the Advanced Driving Assistance System (ADAS) with great progress in recent years. However, for the ADAS, we not only need to detect the behavior of pedestrians in front of the vehicle but also predict future action and the motion trajectory. Therefore, in this paper, we propose a human key point combined optical flow network (KPOF-Net) in the vehicle ADAS for the occlusion situation in the actual scene. When the vehicle encounters a blocked pedestrian at a traffic intersection, we used self-flow to estimate the global optical flow in the image sequence and then proposed a White Edge Cutting (WEC) algorithm to remove obstructions and simply modified the generative adversarial network to initialize pedestrians behind the obstructions. Next, we extracted pedestrian optical flow information and human joint point information in parallel, among which we trained four human key point models suitable for traffic intersections. At last, KPOF-GPDM fusion was proposed to predict the future status and walking trajectories of pedestrians, which combined optical flow information with human key point information. In the experiment, we did not merely compare our method with other four representative approaches in the same scene sequences. We also verified the accuracy of the pedestrian motion state and motion trajectory prediction of the system after fusion of human joint points and optical flow information. Taking into account the real-time performance of the system, in the low-speed and barrier-free environment, the comparative analysis only uses optical flow information, human joint point information, and KPOF-Net three prediction models. The results show that (1) in the same traffic environment, our proposed KPOF-Net can predict the change of pedestrian motion state about 5 frames (about 0.26 s) ahead of other excellent systems; (2) at the same time, our system predicts the trajectory of the pedestrian more accurately than the other four systems, which can achieve more stable minimum error ±0.04 m; (3) in a low-speed, barrier-free experimental environment, our proposed trajectory prediction model that integrates human joint points and optical flow information has higher prediction accuracy and smaller fluctuations than a single-information prediction model, and it can be well applied to automobiles’ ADAS.

Numerous surveillance data processing is crucial in the Internet-of-Things systems with pervasive edge computing. In this process, salient object detection from surveillance videos plays an important role because it provides the human-concerned semantic cue for various industrial tasks. However, it is still challenging for the existing studies with two aspects. The first one is the redundant saliency information from moving background to disturb the detection of salient objects. The second one is the difficulty to model the spatiotemporal saliency uncertainty. To overcome these challenges. In this article, an intelligent approach is proposed for surveillance saliency detection. It enables a region-proposal-based optical flow strategy to suppress the saliency enhancement of non-salient regions due to the moving background. Besides, it develops the bidirectional Bayesian state transition strategy to model the motion uncertainty for refining the spatiotemporal saliency feature. Extensive experiments have been performed on two datasets (the increase of $F_\beta$ is larger than 0.01 for DAVIS, and larger than 0.015 for UVSD), and the comparison with seven methods to evaluate the effectiveness of the proposed approach.

Indirect method is widely used in the field of visual SLAM at present, and it can be divided into feature matching method and optical flow tracking method according to whether matching descriptors are needed in the tracking process. Compared to feature matching method, optical flow does not need to spend too much time on calculating and matching feature points, but it is easy to be affected by illumination changes and fast motion. In recent years, the visual-inertial fusion method improves the localization accuracy and robustness on the condition that the illumination is poor or image blur caused by fast motion. A robust optical flow tracking method based on prediction model for visual-inertial odometry is proposed in this letter. A new optical flow and brightness correction method based on a linear illumination model is used to deal with the large changes of illumination. We also proposed an initial optical flow prediction method with the pre-integration of IMU and reprojection model between image frames to make the optical flow more robust. The results of optical flow are also used to judge the image quality of specific areas to make sure the extracted feature points are more stable and suitable for tracking. Finally, the method is proved that can outperform other state-of-the-art VIO methods in experiments on the public benchmark even without loop closure.

In order to protect the copyright of the high dynamic range (HDR) video, a robust HDR video watermarking method based on saliency extraction and tensor-singular value decomposition (T-SVD) is proposed. Since T-SVD can not only represent high-dimension data, but also remain its intrinsic structure, each frame of the HDR video is considered as the third-order tensor to be transformed by using T-SVD for preserving the main characteristics of the frame. Each frame is divided into non-overlapping blocks, and each block is decomposed by using T-SVD to obtain the orthogonal tensor $${\mathcal{U}}$$ , which includes three orthogonal matrices and represents main energies of the frame. Since the second matrix has more correlations of the video frame than other two matrices, it is used to embed watermark for robustness. Moreover, to obtain the trade-off between watermarking robustness and the visual quality, the saliency map of each frame is extracted to predict the most relevant and important areas for determining the watermark embedding strength. The saliency map is computed based on fusing the spatial saliency and the temporal saliency, where the spatial saliency is built by calculating color, intensity and orientation features of the HDR video and the temporal saliency is obtained by using the optical flow. Experiment results show that the proposed watermarking method can resist a variety of tone mapping attacks and video attacks, and is more robust than existing watermarking methods.

The motion of the camera in a video effectively conveys to the viewers the intention of the director, and is an essential element that enhances their interest. Therefore, detecting the motion of the camera is a very important factor in movie analysis. Existing research to detect the motion of the camera in a video has mainly focused on pan, tilt, and zoom. However, movies use more diverse camera motions to represent complex and varied emotions. Recognizing only pan, tilt, and zoom in a movie has limitations, especially not being able to detect lateral and longitudinal movements of the camera. In this study, a method is proposed to additionally detect boom and truck as well as pan, tilt, and zoom by using deep learning technology to improve this recognition ability. Thus, this study proposes the Improved Extractor of Camera Motion along with the CNN-Based Detector. The Improved Extractor of Camera Motion uses optical flow to extract camera motion vectors from video at eight-frame intervals. The CNN-Based Detector identifies five camera motions by using ResNet-152. As a result, the performance of our proposed method shows accuracy of 86.2%.

Hajj is an annual Islamic pilgrimage, which is attended by millions of pilgrims every year. Therefore, there are many security management problems. The existing solutions can only solve the problems of a small-scale crowd, which contains a simple abnormal behavior and a clear surveillance video. However, the performance hasn’t reached a satisfactory result for a large-scale crowd. Therefore, we propose an abnormal behavior detection method based on optical flow and generative adversarial network (GAN). There are three main contributions in this paper. Firstly, the dynamic features of the model are extracted based on the optical flows. The effectiveness of the features is validated by experiments. Secondly, we propose an optical flow framework based on GAN and use a transfer learning strategy to detect behavioral abnormalities in large-scale crowd scenes. The framework uses U-Net and Flownet to generate and distinguish the normal and abnormal behaviors of individuals within the massive crowds. Finally, a number of abnormal behavior pilgrimage videos from different scenes is collected and tested. The accuracy of UMN scenes 1, 2, 3, and UCSD reaches 99.4%, 97.1%, 97.6% and 89.26%, respectively. It also achieves 79.63% of detection accuracy in the large-scale crowd videos using Abnormal Behaviors HAJJ dataset.

The effect is tested in various specific scenes of sports videos to complete the multitarget motion multitarget tracking detection application applicable to various specific scenes within sports videos. In this paper, deep neural networks are applied to sports video multitarget motion shadow suppression and accurate tracking to improve tracking performance. After the target frame selection is determined, the tracker uses an optical flow method to estimate the limits of the target sports video multitarget motion based on the sports video multitarget motion of the target object between frames. The detector first scans each sports video image frame one by one, observing the previously discovered and learned image frame subregions one by one until the current moment that is highly like the target to be tracked. The preprocessed remote sensing images are converted into grayscale images, the histogram is normalized, and the appropriate height threshold is selected in combination with the regional growth function to realize the rejection of sports video multitarget motion shadow and establish the sports video multitarget network model. The distance and direction of the precise target displacement are determined by frequency-domain vectors and null domain vectors, and the target action judgment mechanism is formed by decision learning. Finally, comparing with other shadow rejection and precision tracking algorithms, the proposed algorithm achieves greater advantages in terms of accuracy and time consumption.

Two-stream CNN is a widely-used network for human action recognition. Two-stream CNN consists of a spatial stream and a temporal stream. The spatial stream, through which the RGB image passes, extracts the shape features of human motion. The temporal stream, through which the optical flow images pass, extracts the sequence features of the listed motions. However, because of the constraints of the optical flow, such as brightness, constancy, and piecewise smoothness, there are limitations to the performance of two-stream CNN. One of the efficient methods to solve this problem is to expand the network model to a three-stream network, fuse it with LSTM, and add a modified pooling layer. This method improves the performance of the model but it increases the computational cost. Besides, the limitations of the optical flow are still present. In this paper, without extending the network model, a binary dense SIFT flow-based two-stream CNN is used instead of the optical flow. Unlike the optical flow, binary dense SIFT flow, which is a feature-based matching flow field is robust in brightness, constancy and piecewise smoothness. To evaluate the binary dense SIFT flow-based two-stream CNN, the UCF-101 dataset was selected for human action recognition. Furthermore, to evaluate the robustness of its brightness constancy and piecewise smoothness, a custom dataset was made up of classes that were extracted from UCF-101. Finally, the proposed method was compared with the state-of-the-art, which uses an optical flow-based two-stream CNN.

Eastern China is one of the most economically developed and densely populated areas in the world. Due to its special geographical location and climate, eastern China is affected by different weather systems, such as monsoons, shear lines, typhoons, and extratropical cyclones. In the near future, the rainfall rate becomes difficult to predict precisely due to these systems. Traditional physics-based methods such as numerical weather prediction (NWP) tend to perform poorly on nowcasting problems due to the spin-up issue. Moreover, various meteorological stations are distributed in this region, generating a large amount of observation data every day, which have great potential for application to data-driven methods. Thus, it is important to train a data-driven model from scratch that is suitable for the specific weather situation of eastern China. However, due to the high degrees of freedom and nonlinearity of machine learning algorithms, it is difficult to add physical constraints. Therefore, with the intention of using various kinds of data as a proxy for physical constraints, we collected three kinds of data (radar, satellite, and precipitation data) in the flood season from 2017 to 2018 in this area and preprocessed them into tensors (256×256) that cover eastern China with a domain of 12.8×12.8∘. The developed multisource data model (MSDM) combines the optical flow, random forest, and convolutional neural network (CNN) algorithms. It treats the precipitation nowcasting task as an image-to-image problem, which takes radar and satellite data with an interval of 30 min as inputs and predicts radar echo intensity with a lead time of 30 min. To reduce the smoothing caused by convolutions, we use the optical flow algorithm to predict satellite data in the following 120 min. The predicted radar echoes from the MSDM together with satellite data from the optical flow algorithm are recursively implemented in the MSDM to achieve a 120 min lead time. The MSDM predictions are comparable to those of other baseline models with a high temporal resolution of 6 min. To solve blurry image problems, we applied a modified structural similarity (SSIM) index as a loss function. Furthermore, we use the random forest algorithm with predicted radar and satellite data to estimate the rainfall rate, and the results outperform those of the traditional, nonlinear radar reflectivity factor and rainfall rate (Z–R) relationships that use logarithmic functions. The experiments confirm that machine learning with multisource data provides more reasonable predictions and reveals a better nonlinear relationship between radar echo and precipitation rate. Apart from developing complicated machine learning algorithms, exploiting the potential of multisource data will yield more improvements.

Background subtraction is an important step involved in solving computer vision problems. This paper proposes a novel background subtraction method with fully residual convolutional neural network (FR-CNN). This fully residual connection helps to fuse the fine scale and coarse scale feature information efficiently. The extracted non-handcrafted features are robust and promisingly efficient compared to the handcrafted features. Furthermore, the method uses temporal and spatial information for the background subtraction process. The optical flow image is used for extracting the temporal information. Additionally, a new background modeling technique is also proposed for the efficient background subtraction. The model is trained using the randomly selected 50 frames from each video sequence of the CDnet-2014 dataset and the FR-CNN model is evaluated by CDnet-2014 dataset. The results shown from the qualitative and quantitative analyses reveal that the FR-CNN model outperforms the state-of-the-art methods.

This paper proposes a new unsupervised domain adaptation approach called Collaborative and Adversarial Network (CAN), which uses the domain-collaborative and domain-adversarial learning strategies for training the neural network. The domain-collaborative learning strategy aims to learn domain specific feature representation to preserve the discriminability for the target domain, while the domain adversarial learning strategy aims to learn domain invariant feature representation to reduce the domain distribution mismatch between the source and target domains. We show that these two learning strategies can be uniformly formulated as domain classifier learning with positive or negative weights on the losses. We then design a collaborative and adversarial training scheme, which automatically learns domain specific representations from lower blocks in CNNs through collaborative learning and domain invariant representations from higher blocks through adversarial learning. Moreover, to further enhance the discriminability in the target domain, we propose Self-Paced CAN (SPCAN), which progressively selects pseudo-labeled target samples for re-training the classifiers. We employ a self-paced learning strategy such that we can select pseudo-labeled target samples in an easy-to-hard fashion. Additionally, we build upon the popular two-stream approach to extend our domain adaptation approach for more challenging video action recognition task, which additionally considers the cooperation between the RGB stream and the optical flow stream. We propose the Two-stream SPCAN (TS-SPCAN) method to select and reweight the pseudo labeled target samples of one stream (RGB/Flow) based on the information from the other stream (Flow/RGB) in a cooperative way. As a result, our TS-SPCAN model is able to exchange the information between the two streams. Comprehensive experiments on different benchmark datasets, Office-31, ImageCLEF-DA and VISDA-2017 for the object recognition task, and UCF101-10 and HMDB51-10 for the video action recognition task, show our newly proposed approaches achieve the state-of-the-art performance, which clearly demonstrates the effectiveness of our proposed approaches for unsupervised domain adaptation.

Artificial intelligence models are widely used in the field of human activity recognition, and human action recognition is an important aspect of human activity recognition. The core of human action recognition is to understand the temporal relationship between video frames. Almost all state-of-the-art methods of human action recognition in videos use optical flow. However, traditional local optical flow estimation methods areexpensive and not trained end-to-end. In this paper, we propose a fast network for human action recognition. Our purpose is to improve the efficiency of optical flow feature extraction and explore the fusion method of spatio-temporal features. For spatio-temporal features, our method combines spatial features and temporal features into fusion features. In addition, we propose CNN with OFF instead of the VGG16 network, which is used to process optical flow features to obtain abundant features. Our model only needs RGB inputs to get the state-of-the-art accuracy of 91.5% on UCF-101, 67.9% on HMDB51, 83.3% on MSR Daily Activity3D, and 91.25% on Florence 3D action, respectively. Compared with most state-of-the-art video action recognition models, our proposed model can effectively improve the accuracy of human action recognition.

First-person video summarization has emerged as an important research problem for computer vision and multimedia communities. In this paper, we show how different graph representations can be developed for accurately summarizing first-person (egocentric) videos in a computationally efficient manner. Each frame in a video is first represented as a weighted graph. A shot boundary detection method using graph based mutual information is developed. We next construct a weighted graph for each shot. A representative frame from each shot is selected using a graph centrality measure. A new way of characterizing egocentric video frames using a graph based center-surround model is shown next. Here, each representative frame is modeled as a union of a center region (graph) and a surround region (graph). By exploiting spectral measures of dissimilarity between the two (center and surround) graphs, optimal center and surround regions are determined. Optimal regions for all frames within a shot are kept the same as that of the representative frame. Center-surround differences in entropy and optical flow values along with PHOG (Pyramidal HOG) features are extracted from each frame. All frames in a video are finally represented by another weighted graph, termed as a Video Similarity Graph (VSG). The frames are clustered by applying a Minimum Spanning Tree (MST) based approach with a new measure for inadmissible edges. Frames closest to the centroid of each cluster are captured to build the summary. Experimental evaluation on two benchmark datasets indicate the advantage of the proposed formulation.

With the improvement of public security awareness, video anomaly detection has become an indispensable demand in surveillance videos. To improve the accuracy of video anomaly detection, this paper proposes a novel two-stream spatial-temporal architecture called Two-Stream Deep Spatial-Temporal Auto-Encoder (Two-Stream DSTAE), which is composed of a spatial stream DSTAE and a temporal stream DSTAE. Firstly, the spatial stream extracts appearance characteristics whereas the temporal stream extracts the motion patterns, respectively. Then, based on the novel policy joint reconstruction error, this model fuses the spatial stream and the temporal stream to extract spatial-temporal characteristics to detect anomalies. Furthermore, since the optical flow is invariant to appearances such as color or light, we introduce optical flow to enhance the capability of extracting continuity between adjacent frames and inter-frame motion information. We demonstrate the accuracy of the proposed method on the publicly available standard datasets: UCSD, Avenue and UMN datasets. Our experiments demonstrate high accuracy, which is superior to the state-of-the-art methods.

Aiming at the problem that it is difficult to balance the speed and accuracy of human behaviour recognition, this paper proposes a method of motion recognition based on random projection. Firstly, the optical flow picture and Red, Green, Blue (RGB) picture obtained by the Lucas-Kanade algorithm are used. Secondly, the data of optical flow pictures and RGB pictures are compressed based on a random projection matrix of compressed sensing, which effectively reduces power consumption. At the same time, based on random projection compression data, it can effectively find the optimal linear representation to reconstruct training samples and test samples. Thirdly, a multichannel 3D convolutional neural network is proposed, and the multiple information extracted by the network is fused to form an output recognizer. Experimental results show that the algorithm in this paper significantly improves the recognition rate of multicategory actions and effectively reduces the computational complexity and running time of the recognition algorithm.

Nowadays, with the increasing number of surveillance cameras, human behavior detection is of importance for public security. Detection of fight behavior using video surveillance is an essential and challenging research field. We propose a multiview fight detection method based on statistical characteristics of the optical flow and random forest. Cyberphysical systems for monitoring can obtain timely and accurate information from this method. Two novel descriptors named Motion Direction Inconsistency (MoDI) and Weighted Motion Direction Inconsistency (WMoDI) are defined to improve the performance of existing methods for videos with different shooting views and solve the misjudgment on nonfight, such as running and talking. First, YOLO V3 algorithm is applied to mark the motion areas, and then, the optical flow is computed to extract descriptors. Finally, Random Forest is used for classification based on statistical characteristics of descriptors. The evaluation results on CASIA dataset demonstrate that the proposed method can improve the accuracy and reduce the rate of missing alarm and false alarm for the detection, and it is very robust against videos with different shooting views.

Zusammenfassung In dieser Arbeit wird eine effiziente Methode zur Schätzung der Ego-Fahrstreifen vorgestellt. Das System ist ausgelegt auf die Anwendung für Mikromobilitätssysteme wie E-Fahrräder, E-Scooter oder E-Dreiräder und basiert auf Sequenzen von RGB-Bildern, die aus der Fahrerperspektive aufgenommen werden. Um die Grenzen verschiedener Fahrbahntypen, insbesondere Fußwegen, Radwegen und Straßen, robust zu detektieren, wird ein gradientenbasierter Kantendetektor mit einem Farbsegmentierungansatz kombiniert. Sowohl geometrische Gegebenheiten als auch zeitliche Inferenz werden genutzt, um die Schätzung zu optimieren, insbesondere in schwierigen Fällen wie Kurven. Dabei ist insbesondere die dynamische Schätzung des Fluchtpunkts mithilfe des Optischen Flusses zwischen zwei aufeinanderfolgenden Bildern von Bedeutung. Tests mit zwei Aufnahmesetups und drei verschiedenen Aufrahmeraten zeigen, dass der Ego-Fahrstreifen in den meisten Fällen mit hoher Genauigkeit detektiert und in Kurven passend approximiert wird. Dies wird von der quantitative Evaluation anhand von über 2000 Testbildern bestätigt, die eine IoU von 75,28 % ergibt. Eine Performanz von 12 fps auf einem Raspberry Pi 3 zeigt zudem, dass die Methode wenig rechenintensiv ist, was als Vorraussetzung für die Anwendung in der Mikromobilität gilt.

Abnormal behaviour detection algorithm needs to conduct behaviour analysis on the basis of continuous video inclination tracking, and the robustness of the algorithm is reduced for the occlusion of moving targets, the occlusion of the environment, and the movement of targets with the same colour. For this reason, the optical flow information between RGB (red, green, and blue) images and video frames is used as the input of the network in view of group behaviour. Then, the direction, velocity, acceleration, and energy of the crowd were weighted and fused into a global optical flow descriptor. At the same time, the crowd trajectory map is extracted from the original image of a single frame. Following, in order to realize the detection of large displacement moving target and solve the problem that the traditional optical flow algorithm is only suitable for the detection of displacement moving target, a video abnormal behaviour detection algorithm based on the double-flow convolutional neural network is proposed. The network uses two network branches to learn spatial dimension information and temporal dimension information, respectively, and uses short- and long-time neural network to model the dependency relationship between long-time video frames, so as to obtain the final behaviour classification results. Simulation test results show that the proposed method can achieve good recognition effect on multiple datasets, and the performance of abnormal behaviour detection can be significantly improved by using interframe motion information.

Automatic video surveillance in public crowded places has been an active research area for security purposes. Traditional approaches try to solve the crowd behavior recognition task using a sequential two-stage pipeline as low-level feature extraction and classification. Lately, deep learning has shown promising results in comparison to traditional methods by extracting high-level representation and solving the problem in an end-to-end pipeline. In this paper, we investigate a deep architecture for crowd event recognition to detect seven behavior categories in PETS2009 event recognition dataset. More especially, we apply an integrated handcrafted and Conv-LSTM-AE method with optical flow images as input to extract a high-level representation of data and conduct classification. After achieving a latent representation of input optical flow image sequences in the bottleneck of autoencoder(AE), the architecture is split into two separate branches, one as AE decoder and the other as the classifier. The proposed architecture is jointly trained for representation and classification by defining two different losses. The experimental results in comparison to the state-of-the-art methods demonstrate that our algorithm can be promising for real-time event recognition and achieves a better performance in calculated metrics.

In the field of additive manufacturing (AM), the implementation of real-time monitoring systems is essential to guarantee the quality and repeatability of the manufactured parts. Particularly in the laser metal deposition technique, the dynamics and stability of the molten pool, both geometrically and thermally, have an impact on the microstructural properties and the formation of defects. This article presents a novel technique to track the molten pool in real-time. A middle wavelength infrared (MWIR) camera is used, with a spectral range between 3 and 5 μm. The acquired frames are processed to reduce their dimension using a new strategy based on Superpixels. Traditional pixels are grouped into block-structures that are similar in temperature, optical flow, and spatial position. This guarantees a better adherence to the edges improving the segmentation task. Then, the optical flow of the molten pool is calculated. Throughout this tracking, geometric and thermal features are extracted. The proposed method was evaluated using qualitative and quantitative criteria. The performance of the proposal is compared against benchmark algorithms using standard metrics such as Sub-segmentation Error, Compactness, and Achievable Segmentation Accuracy obtaining better results. The proposal presented is robust to the number of superpixels generated as well as the size of the images. Which provides better performance for in situ monitoring tasks. Our proposal has significant industrial potential for monitoring additive manufacturing processes, as it requires minimal modifications to commercially available industrial machines.

The increased use of UAVs (Unmanned Aerial Vehicles) has heightened demands for an automated landing system intended for a variety of tasks and emergency landings. A key challenge of this system is finding a safe landing site in an unknown environment using on-board sensors. This paper proposes a method to generate a heat map for safety evaluation using images from a single on-board camera. The proposed method consists of the classification of ground surface by CNNs (Convolutional Neural Networks) and the estimation of surface flatness from optical flow. We present the results of applying this method to a video obtained from an on-board camera and discuss ways of improving the method.

Micro-expression recognition is a substantive cross-study of psychology and computer science, and it has a wide range of applications (e.g., psychological and clinical diagnosis, emotional analysis, criminal investigation, etc.). However, the subtle and diverse changes in facial muscles make it difficult for existing methods to extract effective features, which limits the improvement of micro-expression recognition accuracy. Therefore, we propose a multi-scale joint feature network based on optical flow images for micro-expression recognition. First, we generate an optical flow image that reflects subtle facial motion information. The optical flow image is then fed into the multi-scale joint network for feature extraction and classification. The proposed joint feature module (JFM) integrates features from different layers, which is beneficial for the capture of micro-expression features with different amplitudes. To improve the recognition ability of the model, we also adopt a strategy for fusing the feature prediction results of the three JFMs with the backbone network. Our experimental results show that our method is superior to state-of-the-art methods on three benchmark datasets (SMIC, CASME II, and SAMM) and a combined dataset (3DB).

Fall detection is drawing serious attention all across the globe, as unattended fall of senior citizens creates long lasting injuries. This necessitates the deployment of automatic fall detection systems to facilitate smart care health environments for the elderly people living in various settings, viz., living independently in their homes, hospitalized or living in care homes. The proposed work employs Siamese network with one shot classification for human fall detection. Unlike the neural network that classifies the video sequences, this network learns to differentiate the video sequences by computing the similarity score. The network contains two identical CNNs, receiving pair of video sequences as the input. The features of these networks are merged at the final layer through the similarity function. Two different architectures viz., one with 2D convolutional filters and the other with depth wise convolutional filters, each operated on two set of features, RGB and optical flow features are developed. Experimental results demonstrate the effectiveness and feasibility of the proposed work compared to state-of-the methods.

With the increasing population, the probability of occurrence of different kinds of crowd anomalies gets frequent. Blockage on roads, the lighting condition, and the uneven movement of humans and vehicles makes it a tough and challenging problem. The paper proposes the combined use of a convolutional neural network and bidirectional LSTM to solve the task. CNN helps extract frame-level features of the optical flow over the video evaluated by the Lucas Kanade algorithm. A novel approach of improving the predicted class with the domain knowledge of datasets is also performed. The proposed methodology is tested on the crowd anomaly dataset's benchmark datasets, namely UCSD Ped-1 and UCSD Ped-2, and it outperforms various other existing state-of-the-art methods.

Recent studies on vision-based tactile sensing have shown promising results on the perception of contact information, which could improve the performance of dexterous manipulations. However, 3-dimensional contact deformation tracking at a higher resolution is desired and remains a challenge for vision-based tactile sensors with monocular configurations. In this work, a similar hardware structure to our previous FingerVision sensor is adopted. The dot markers are replaced with a novel random color pattern as the sensor's tracking target and a dense optical flow algorithm is used to track the deformation of its elastic contact interface. This results in a more accurate 2-dimensional deformation field estimation at a higher resolution in comparison with that obtained using sparse dot markers. Additionally, the denser and more accurate deformation field enables depth estimation with better fidelity. To achieve depth estimation purely from the optical flow field, Gaussian density feature extraction and processing framework are proposed. The resulting depth map can be used independently as a tactile sensing modality, or jointly with the accurate in-plane displacement field as a more complete deformation tracking of contact interfaces for vision-based tactile sensors.

One of the problems solved by analyzing the data of long-term Video EEG monitoring is the differentiation of epileptic and artifact events. For this, not only multichannel EEG signals are used, but also video data analysis, since traditional methods based on the analysis of EEG wavelet spectrograms cannot reliably distinguish an epileptic seizure from a chewing artifact. In this paper, we propose an algorithm for detecting artifact events based on a joint analysis of the level of the optical flow and the ridges of wavelet spectrograms. The preliminary results of the analysis of real clinical data are given. The results show the possibility in principle of reliable distinguishing non-epileptic events from epileptic seizures.

For a long time, learning spatiotemporal features with deep neural networks has been a difficult task in the field of computer vision. In this paper, we present a novel deep architecture, termed as Bifurcated Convolutional Neural Network (BifurcatedNet) to learn the discriminative video representation in an end-to-end manner. In our work, the BifurcatedNet is built upon the stacking bifurcated blocks that aim at simultaneously capturing the static appearance information and the temporal dynamic from input data. Specifically, the bifurcated block is composed of two separated branch, i.e., an appearance branch and a dynamic branch. The appearance branch employs 2D convolutional operation to obtain the spatial responses of image pixels or filters of each input frame, while the design of the dynamic branch is based on the spatio-temporal convolutional operation to exploit the temporal dynamic between pixels and filter response across multiple frames. Multiple experiments are conducted on two popular action recognition benchmarks: UCF101 and HMDB51. With only RGB input, the BifurcatedNet obtains the superior performance over the existing state-of-the-art models under the same experimental setting. The proposed BifurcatedNet is also implemented in a two-stream fashion by using both RGB and optical flow input, and still achieves the state-of-the-art performance, demonstrating the effectiveness of the network design. Furthermore, in order to evaluate the generalization ability, we conduct experiments on the Chalearn LAP IsoGD dataset and find that our model works well in gesture recognition tasks.

This paper presents a proposed multistage deep neural network pipeline for sports action recognition. The proposed pipeline is based on the classification of stroke types of table tennis using spatiotemporal features. The proposed network predicts the final class with different aspects of the final class at each stage. Outcomes of each stage are then fused together to obtain the final prediction. We utilize four different methods that are used in each stage, namely RGB image-based, optical flow-based, pose-based, and region-of-interest-based methods. We conducted our experiments on the TTSTROKE-21 dataset, which has been introduced in MediaEval Challenge 2020. Experimental results show that our proposed methodology obtains 90.7% test accuracy using a combination of RGB images and optical flow-based methods together.

Whole pixel registration of non-rigid images with high accuracy and efficiency is a challenging problem in computer vision. To address this issue, we propose a correspondence vector field (CVF) Interpolation approach based on sparse matching of feature seeds. First, we detect and match two types of feature seeds to improve the accuracy of the later dense CVF interpolation. The first type of feature seeds is to guarantee the accuracy of the motion boundary, while the second one is to achieve the uniform distribution of seeds, which is helpful to improve the effect of interpolation. Second, we regionally estimate the dense CVF using the proposed interpolation approach on this basis. At last, we realize the whole-pixel registration of non-rigid images to yield the image alignment. Different from the traditional CVF interpolation approaches based on optical flow field, ours is based on the sparse matching of feature seeds. Thus, it is not limited to the large displacements and tends to achieve the accurate matching of certain key points easily, which is critical to the final interpolation result. Qualitative and quantitative experimental results on several internationally used datasets demonstrate that our approach outperforms the state-of-the-art ones.

Computer vision is widely used in manufacturing, sports, medical diagnosis, and other fields. In this article, a multifeature fusion error action expression method based on silhouette and optical flow information is proposed to overcome the shortcomings in the effectiveness of a single error action expression method based on the fusion of features for human body error action recognition. We analyse and discuss the human error action recognition method based on the idea of template matching to analyse the key issues that affect the overall expression of the error action sequences, and then, we propose a motion energy model based on the direct motion energy decomposition of the video clips of human error actions in the 3 Deron action sequence space through the filter group. The method can avoid preprocessing operations such as target localization and segmentation; then, we use MET features and combine with SVM to test the human body error database and compare the experimental results obtained by using different feature reduction and classification methods, and the results show that the method has the obvious comparative advantage in the recognition rate and is suitable for other dynamic scenes.

Automated human posture estimation (A-HPE) systems need delicate methods for detecting body parts and selecting cues based on marker-less sensors to effectively recognize complex activity motions. Recognition of human activities using vision sensors is a challenging issue due to variations in illumination conditions and complex movements during the monitoring of sports and fitness exercises. In this paper, we propose a novel A-HPE method that intelligently identifies human behaviours by utilizing saliency silhouette detection, robust body parts model and multidimensional cues from full-body silhouettes followed by an entropy Markov model. Initially, images are pre-processed and noise is removed to obtain a robust silhouette. Body parts models are then used to extract twelve key body parts. These key body parts are further optimized to assist the generation of multidimensional cues. These cues include energy, optical flow and distinctive values that are fed into quadratic discriminant analysis to discriminate cues which help in the recognition of actions. Finally, these optimized patterns are further processed by a maximum entropy Markov model as a recognizer engine based on transition and emission probability values for activity recognition. For evaluation, we used a leave-one-out cross validation scheme and the results outperformed existing well-known statistical state-of-the-art methods by achieving better body parts detection and higher recognition accuracy over four benchmark datasets. The proposed method will be useful for man-machine interactions such as 3D interactive games, virtual reality, service robots, e-health fitness, and security surveillance.

In the computer vision applications such as security surveillance and robotics, pedestrian identification shows much attention in the last decade. This is usually achieved by human biometrics. Besides human biometrics, sometimes it is required to identify pedestrians at a distance. This could be accomplished based on a fact of different whole-body appearances. The real-time pedestrian identification is a challenging task due to several factors such as illumination effects, noise, change in viewpoint, and video resolution. The more recent, the deep neural network (DNN) shows a massive performance for various real-world applications. In this article, we present a real-time architecture for pedestrian identification using motion-controlled DNN. In the proposed architecture, the motion vectors are calculating using optical flow and then utilized in the next step, named features extraction. Two types of features, such as HOG and DNN, are computing. The pre-trained VGG19 CNN model is employing and trained through transfer learning. The deep learning features are extracted from two layers—fully connected layers 7 and 8. Also, we proposed a feature selection method named Bayesian modeling along with LSVM. The best selected features of both HOG and DNN are finally fused in one matrix for final identification. The multi-class support vector machine classifier is used for final identification. The videos are recording in the real-time environment for the experimental process and achieve an average accuracy of 98.62%. Overall, identification accuracy shows the effectiveness of the proposed approach.

Recent advances in video frame interpolation have shown that convolutional neural networks combined with optical flow are capable of producing a high-quality intermediate frame between two consecutive input frames in most scenes. However, existing methods have difficulties dealing with large and non-uniform motions that widely exist in real-world scenes because they often adopt the same strategy to deal with different motions, which easily results in unsatisfactory results. In this article, we propose a novel fine-grained motion estimation approach (FGME) for video frame interpolation. It mainly contains two strategies: multi-scale coarse-to-fine optimization and multiple motion features estimation. The first strategy is to gradually refine optical flows and weight maps, both of which are used to synthesize the target frame. The second strategy aims to provide fine-grained motion features by generating multiple optical flows and weight maps. To demonstrate its effectiveness, we propose a fully convolutional neural network with three refinement scales and four motion features. Surprisingly, this simple network produces state-of-the-art results on three standard benchmark datasets and real-world examples, with advantages in terms of effectiveness, simplicity, and network size over other existing approaches. Furthermore, we demonstrate that the FGME approach has good generality and can significantly improve the synthesis quality of other methods.

We propose a novel method of abnormal crowd behavior detection in surveillance videos. Mainly, our work focuses on detecting crowd divergence behavior that can lead to serious disasters like a stampede. We introduce a notion of physically capturing motion in the form of images and classify crowd behavior using a convolution neural network (CNN) trained on motion-shape images (MSIs). First, the optical flow (OPF) is computed, and finite-time Lyapunov exponent (FTLE) field is obtained by integrating OPF. Lagrangian coherent structure (LCS) in the FTLE field represents crowd-dominant motion. A ridge extraction scheme is proposed for the conversion of LCS-to-grayscale MSIs. Lastly, a supervised training approach is utilized with CNN to predict normal or divergence behavior for any unknown image. We test our method on six real-world low- as well as high-density crowd datasets and compare performance with state-of-the-art methods. Experimental results show that our method is not only robust for any type of scene but also outperform existing state-of-the-art methods in terms of accuracy. We also propose a divergence localization method that not only identifies divergence starting (source) points but also comes with a new feature of generating a ‘localization mask’ around the diverging crowd showing the size of divergence. Finally, we also introduce two new datasets containing videos of crowd normal and divergence behaviors at the high density.

The problem of action and interaction recognition of human activities from the perspective of first-person view-point is an interesting area of research in the field of human action recognition (HAR). This paper presents a data-driven spatio-temporal network to combine different modalities computed from first-person videos using a temporal attention mechanism. First, our proposed approach uses three-stream inflated 3D ConvNet (I3D) to extract low-level features from RGB frame difference (FD), optical flow (OF) and magnitude-orientation (MO) streams. An I3D network has the advantage to directly learn spatio-temporal features over short video snippets (like 16 frames). Second, the extracted features are fused together and fed to a Bidirectional long short-term memory (BiLSTM) network to model high-level temporal feature sequences. Third, we propose to incorporate attention mechanism with our BiLSTM network to automatically select the most relevant temporal snippets in the given video sequence. Finally, we conducted extensive experiments and achieve state-of-the-art results on JPL (98.5%), NUS (84.1%), UTK (91.5%) and DogCentric (83.3%) datasets. These results show that features extracted from three-stream network are complementary to each other, and attention mechanism further improves the results by a large margin than previous attempts based on handcrafted and deep features.

We propose a novel algorithm for stabilizing selfie videos. Our goal is to automatically generate stabilized video that has optimal smooth motion in the sense of both foreground and background. The key insight is that non-rigid foreground motion in selfie videos can be analyzed using a 3D face model, and background motion can be analyzed using optical flow. We use second derivative of temporal trajectory of selected pixels as the measure of smoothness. Our algorithm stabilizes selfie videos by minimizing the smoothness measure of the background, regularized by the motion of the foreground. Experiments show that our method outperforms state-of-the-art general video stabilization techniques in selfie videos.

Each normal human has pairwise hands, which can make sophisticated hand gestures delivering different semantic meanings. In practice, disabled persons can communicate with each other by their hand gestures. However, for normal people without specific training process, it is challenging for them to understand the meaning of the various hand gestures. In this work, we propose a novel quality-aware human–computer interaction (HCI) framework for understanding the sophisticated human hand gestures, wherein the key technique is a multi-view feature learning algorithm that optimally fuses hand silhouettes, different figure positions, and hand motions. More specifically, given each human hand, we first extract the multimodal visual features, such as the hand silhouettes based on background removal, different figure position localization using active learning, and hand motions by leveraging optical flow. Afterward, we propose a multi-view learning-based feature fusing scheme that optimizes the multimodal features both locally and globally. Based on this, the optimal weights of different features channels can be calculated. By leveraging the optimally fused feature, we train a multi-class kernel machine to classify the various human hand gestures into each category with a particular meaning. Comprehensive experimental results on a large-scale human hand gesture data set have demonstrated that our method can achieve a highly competitive recognition accuracy by spending less time consumption. Besides, our method can robustly support arbitrary number of semantic category characterizing human hand gestures. Last but not least, our proposed hand gesture understanding technique can be conveniently incorporated into many state-of-the-art HCI system.

Stereoscopic video quality assessment has become a major research topic in recent years. Existing stereoscopic video quality metrics are predominantly based on stereoscopic image quality metrics extended to the time domain via for example temporal pooling. These approaches do not explicitly consider the motion sensitivity of the Human Visual System (HVS). To address this limitation, this paper introduces a novel HVS model inspired by physiological findings characterising the motion sensitive response of complex cells in the primary visual cortex (V1 area). The proposed HVS model generalises previous HVS models, which characterised the behaviour of simple and complex cells but ignored motion sensitivity, by estimating optical flow to measure scene velocity at different scales and orientations. The local motion characteristics (direction and amplitude) are used to modulate the output of complex cells. The model is applied to develop a new type of full-reference stereoscopic video quality metrics which uniquely combine non-motion sensitive and motion sensitive energy terms to mimic the response of the HVS. A tailored two-stage multi-variate stepwise regression algorithm is introduced to determine the optimal contribution of each energy term. The two proposed stereoscopic video quality metrics are evaluated on three stereoscopic video datasets. Results indicate that they achieve average correlations with subjective scores of 0.9257 (PLCC), 0.9338 and 0.9120 (SRCC), 0.8622 and 0.8306 (KRCC), and outperform previous stereoscopic video quality metrics including other recent HVS-based metrics.