Search

With the goal of solving the problem of feature extractors lacking strong supervision training and insufficient time information concerning single‐sequence model learning, a hierarchical sequence memory network with a multi‐level iterative optimisation strategy is proposed for continuous sign language recognition. This method uses the spatial‐temporal fusion convolution network (STFC‐Net) to extract the spatial‐temporal information of RGB and Optical flow video frames to obtain the multi‐modal visual features of a sign language video. Then, in order to enhance the temporal relationships of visual feature maps, the hierarchical memory sequence network is used to capture local utterance features and global context dependencies across time dimensions to obtain sequence features. Finally, the decoder decodes the final sentence sequence. In order to enhance the feature extractor, the authors adopted a multi‐level iterative optimisation strategy to fine‐tune STFC‐Net and the utterance feature extractor. The experimental results on the RWTH‐Phoenix‐Weather multi‐signer 2014 dataset and the Chinese sign language dataset show the effectiveness and superiority of this method. [ABSTRACT FROM AUTHOR]

In recent years, the analysis of macro- and micro-expression has drawn the attention of researchers. These expressions provide visual cues to an individual's emotions, which can be used in a broad range of potential applications such as lie detection and policing. In this paper, we address the challenge of spotting facial macro- and micro-expression from videos and present compelling results by using a deep learning approach to analyze the optical flow features. Unlike other deep learning approaches that are mainly based on Convolutional Neural Networks (CNNs), we propose a Transformer-based deep learning approach that predicts a score indicating the probability of a frame being within an expression interval. In contrast to other Transformer-based models that achieve high performance by being pre-trained on large datasets, our deep learning model, called SL-Swin, which incorporates Shifted Patch Tokenization and Locality Self-Attention into the backbone Swin Transformer network, effectively spots macro- and micro-expressions by being trained from scratch on small-size expression datasets. Our evaluation outcomes surpass the MEGC 2022 spotting baseline result, obtaining an overall F1-score of 0.1366. Additionally, our approach performs well on the MEGC 2021 spotting task, with an overall F1-score of 0.1824 and 0.1357 on the CAS(ME) 2 and SAMM Long Videos, respectively. The code is publicly available on GitHub. [ABSTRACT FROM AUTHOR]

The detection of moving objects in images is a crucial research objective; however, several challenges, such as low accuracy, background fixing or moving, 'ghost' issues, and warping, exist in its execution. The majority of approaches operate with a fixed camera. This study proposes a robust feature threshold moving object identification and segmentation method with enhanced optical flow estimation to overcome these challenges. Unlike most optical flow Otsu segmentation for fixed cameras, a background feature threshold segmentation technique based on a combination of the Horn–Schunck (HS) and Lucas–Kanade (LK) optical flow methods is presented in this paper. This approach aims to obtain the segmentation of moving objects. First, the HS and LK optical flows with the image pyramid are integrated to establish the high-precision and anti-interference optical flow estimation equation. Next, the Delaunay triangulation is used to solve the motion occlusion problem. Finally, the proposed robust feature threshold segmentation method is applied to the optical flow field to attract the moving object, which is the. extracted from the Harris feature and the image background affine transformation model. The technique uses morphological image processing to create the final moving target foreground area. Experimental results verified that this method successfully detected and segmented objects with high accuracy when the camera was either fixed or moving. [ABSTRACT FROM AUTHOR]

As image acquisition devices have outstanding potential for gathering vibration information, computer vision has received a lot of interest in structural health monitoring (SHM). In this work, a fully automated peak picking methodology based on computer vision in tandem with deep learning is proposed to realize vibration measurements and identify natural frequencies from the plot of the power spectral density transmissibility (PSDT). A deep-learning-enhanced image processing technology was used to extract the vibration signals with automatic active pixel selection, while a convolutional neural network was used to further process the vibration measurements so that the frequencies could be identified from PSDT-based functions. The proposed method was verified by three case studies, including the dynamic testing of two laboratory models and the field testing of the stay cable. The findings showed that the proposed deep-learning-enhanced approach has a high potential for use in SHM by automatically performing vibration measurement and frequency extraction. [ABSTRACT FROM AUTHOR]

This study sought an effective detection method not only for flame but also for the smoke generated in the event of a fire. To this end, the flame region was pre-processed using the color conversion and corner detection method, and the smoke region could be detected using the dark channel prior and optical flow. This eliminates unnecessary background regions and allows selection of fire-related regions. Where there was a pre-processed region of interest, inference was conducted using a deep-learning-based convolutional neural network (CNN) to accurately determine whether it was a flame or smoke. Through this approach, the detection accuracy is improved by 5.5% for flame and 6% for smoke compared to when a fire is detected through the object detection model without separate pre-processing. [ABSTRACT FROM AUTHOR]

Optical flow estimation is a fundamental task in computer vision and image processing. Due to the difficulty in obtaining the ground truth of flow field, unsupervised learning approaches attract more and more research interests in recent years. However, despite of their good generalization capability, unsupervised optical flow methods suffer in the scenarios with large displacement, small objects, and occlusions. In this work, we propose a novel optical flow network based on decoder with multi-scale kernels. Different from previous U-Net like or pyramidal methods, we design our network based on RAFT architecture that with a 4D correlation layer and recurrent decoder. More importantly, we incorporate three novel ideas with regard to the input, information processing and output of the update units improve the performance. Firstly, we utilize various motion-related information as input to the update units. Secondly, we propose a module of multi-scale update unit. Thirdly, for the final flow up-sampling procedure, we propose an image-guided up-sampling loss to guide the learning of up-sampling masks. Our model is trained by the occlusion-aware photometric loss, edge-aware smoothness loss, self-supervised loss, and image-guided up-sampling loss. Experimental results demonstrate that our model achieves the state-of-the-art performance on both Sintel and KITTI and outperforms other unsupervised optical flow methods remarkably. [ABSTRACT FROM AUTHOR]

Fractional calculus is an abstract idea exploring interpretations of differentiation having non-integer order. For a very long time, it was considered as a topic of mere theoretical interest. However, the introduction of several useful definitions of fractional derivatives has extended its domain to applications. Supported by computational power and algorithmic representations, fractional calculus has emerged as a multifarious domain. It has been found that the fractional derivatives are capable of incorporating memory into the system and thus suitable to improve the performance of locality-aware tasks such as image processing and computer vision in general. This article presents an extensive survey of fractional-order derivative-based techniques that are used in computer vision. It briefly introduces the basics and presents applications of the fractional calculus in six different domains viz. edge detection, optical flow, image segmentation, image de-noising, image recognition, and object detection. The fractional derivatives ensure noise resilience and can preserve both high and low-frequency components of an image. The relative similarity of neighboring pixels can get affected by an error, noise, or non–homogeneous illumination in an image. In that case, the fractional differentiation can model special similarities and help compensate for the issue suitably. The fractional derivatives can be evaluated for discontinuous functions, which help estimate discontinuous optical flow. The order of the differentiation also provides an additional degree of freedom in the optimization process. This study shows the successful implementations of fractional calculus in computer vision and contributes to bringing out challenges and future scopes. [ABSTRACT FROM AUTHOR]

In this article, we propose a novel model for facial micro-expression (FME) recognition. The proposed model basically comprises a transformer, which is recently used for computer vision and has never been used for FME recognition. A transformer requires a huge amount of data compared to a convolution neural network. Then, we use motion features, such as optical flow and late fusion to complement the lack of FME dataset. The proposed method was verified and evaluated using the SMIC and CASME II datasets. Our approach achieved state-of-the-art (SOTA) performance of 0.7447 and 73.17% in SMIC in terms of unweighted F1 score (UF1) and accuracy (Acc.), respectively, which are 0.31 and 1.8% higher than previous SOTA. Furthermore, UF1 of 0.7106 and Acc. of 70.68% were shown in the CASME II experiment, which are comparable with SOTA. [ABSTRACT FROM AUTHOR]

In the field of computer vision, high-dynamic dance motion recognition is a difficult problem to solve. Its goal is to recognize human motion by analyzing video data using image processing and classification recognition technology. Video multifeature fusion has sparked a surge in research in a variety of fields. Several pixel points that can be distinguished and displayed in several adjacent images that can reflect their characteristics are referred to as multifeature fusion. It is responsible for a significant portion of the similarity results between the two video segments. Motion recognition relies heavily on video multifeature fusion, which has a direct impact on the robustness and accuracy of recognition results. The directional gradient histogram features, optical flow direction histogram features, and audio features extracted from dance video are used to characterize dance movements after all of the characteristics of dance movements have been considered. This paper focuses on the high-dynamic dance action recognition method based on video multifeature fusion, which aims to combine high-dynamic dance action recognition and video multifeature fusion. [ABSTRACT FROM AUTHOR]

[Display omitted] • Vision-based cutting tool motion registration methods are presented. • Tool motion is recorded with low- and high-speed cameras with high resolutions. • Motion is estimated using edge detection, optical flow, and DIC schemes. • Tool motion from vision-based measurements agree with integrated accelerations. • Modal parameters from vision-based response compare well with standard procedures. This paper presents the use of vision-based methods for cutting tool motion registration and modal analysis. Motion of three illustrative tools were recorded using low- and high-speed cameras with sufficiently high resolutions. The tool's own features are used to register motion. Pixels within images from recordings of the vibrating tools are treated as non-contact motion sensors. Comparative analysis of three different methods of motion registration are presented to evaluate their suitability for the application of interest. These include variants of expanded edge detection and tracking schemes, expanded optical flow-based schemes, and established digital image correlation methods. Performance of different methods was observed to be governed by the tool's own features, illumination conditions, noise, and the image acquisition parameters. Extracted motion was benchmarked against twice integrated measured tool point accelerations, and motion was generally observed to compare well. Modal parameters extracted from vision-based measurements were also observed to agree with those extracted using more traditional experimental modal analysis procedures using a contact type accelerometer as the transducer. Since methods presented are generalized, they can suitably be adapted for other applications of interest. [ABSTRACT FROM AUTHOR]

Current vision-based displacement measurement methods have limitations such as being in need of manual targets and parameter adjustment, and significant user involvement to reach the desired result. This study proposes a novel structural displacement measurement method using deep learning-based full field optical flow methods. The performance of the proposed method is verified via a laboratory experiment conducted on a grandstand structure with a comparative study, where the same data samples are analysed with a commonly used vision-based method, and a displacement sensor measurement is used as the ground truth. Statistical analysis of the comparative results show that the proposed method gives higher accuracy than the traditional optical flow algorithm and shows consistent results in compliance with displacement sensor measurements. Image collection, tracking, and non-uniform sampling are investigated in the experimental data and suggestions are made to obtain more accurate displacement measurements. A field-validation on a footbridge showed that the measurement error induced by the camera motion is mitigated by a camera motion subtraction procedure. The proposed method has good potential to be applied by structural engineers, who have little or no experience in computer vision and image processing, to do vision-based displacement measurements. [ABSTRACT FROM AUTHOR]

In a photo, motion blur can be used as an artistic style to convey motion and to direct attention. In panning or tracking shots, a moving object of interest is followed by the camera during a relatively long exposure. The goal is to get a blurred background while keeping the object sharp. Unfortunately, it can be difficult to impossible to precisely follow the object. Often, many attempts or specialized physical setups are needed. This paper presents a novel approach to create such images. For capturing, the user is only required to take a casually recorded hand‐held video that roughly follows the object. Our algorithm then produces a single image which simulates a stabilized long time exposure. This is achieved by first warping all frames such that the object of interest is aligned to a reference frame. Then, optical flow based frame interpolation is used to reduce ghosting artifacts from temporal undersampling. Finally, the frames are averaged to create the result. As our method avoids segmentation and requires little to no user interaction, even challenging sequences can be processed successfully. In addition, artistic control is available in a number of ways. The effect can also be applied to create videos with an exaggerated motion blur. Results are compared with previous methods and ground truth simulations. The effectiveness of our method is demonstrated by applying it to hundreds of datasets. The most interesting results are shown in the paper and in the supplemental material. [ABSTRACT FROM AUTHOR]

Abstract Dense motion estimations obtained from optical flow techniques play a significant role in many image processing and computer vision tasks. Remarkable progress has been made in both theory and its application in practice. In this paper, we provide a systematic review of recent optical flow techniques with a focus on the variational method and approaches based on Convolutional Neural Networks (CNNs). These two categories have led to state-of-the-art performance. We discuss recent modifications and extensions of the original model, and highlight remaining challenges. For the first time, we provide an overview of recent CNN-based optical flow methods and discuss their potential and current limitations. Highlights • Introducing optical flow: the basic concepts, the characteristics of the variational and CNN-based techniques, and the evaluation measures. • Discussing developments of the variational method, analyzing the challenges and illustrating the corresponding treating strategies of it. • Describing the conception of the CNN-based technique, and give a detailed discussion of the issues of this technique. [ABSTRACT FROM AUTHOR]

Micro-expression analysis is an interesting and challenging task in computer vision. It has inspired a series of possible applications in many areas such as police–criminal interrogation and important business negotiation. One of the most crucial step in a micro-expression recognition system is the extraction of well discriminating features. In this paper, we propose a new feature based on the fusion of motion boundary histograms (FMBH). This feature is generated by combining both the horizontal and the vertical components of the differential of optical flow as inspired from the motion boundary histograms (MBH). The proposed feature is then validated and evaluated through the leave-one-subject-out (LOSO) protocol for micro-expression recognition. Moreover, the proposed method is compared to state-of-the-art methods on four well-known databases CASME, CASME II, SMIC and CAS(ME) 2 . Comparative experimental results demonstrate that the proposed FMBH feature descriptor yields promising performance. [ABSTRACT FROM AUTHOR]

We focus on the challenging cross-view action recognition problem. The key to this problem is to find the correspondence between two different views, which is realized in two stages. First, we construct a dualcodebook for the two views, which contains one codebook for each view. Each codeword in one codebook has a corresponding codeword in the other codebook, whereas traditional methods implement independent codebooks for the views. We propose an effective coclustering algorithm based on seminonnegative matrix factorization to derive the dual-codebook. Additionally, to represent actions in one view, unlike most other methods using the codebook of that view only, we also exploit the codebook-specific information from the other view. Thus, we construct mapped-codebooks via codebook transformation, complementing the codebook-to-codebook correspondence within the dual-codebook. In the second stage, observing that the temporal relationship between action segments within an action is view invariant, we further propose a hierarchical transfer framework based on a temporal structure that can effectively capture such action-segment temporal relationship at multiple timescales, which is more discriminative than the usual video-level transfer strategy. Extensive experimental results on the INRIA xmas motion acquisition sequences and West Virginia University datasets demonstrate superiority of the proposed method compared with state-of-the-art approaches. ©2018 SPIE and IS&T [DOI: 10 .1117/1JEI.27.4.043044] [ABSTRACT FROM AUTHOR]

The detection of foreign substances in transparent-bottled liquid is a key procedure in production. An improved algorithm based on the combination of guided filter and visual background extractor (ViBe) is proposed for the detection of foreign substances in transparent-bottled turbid liquid on the market. In the stage of image preprocessing, the image enhancement is carried out by the guided filter and then the liquid region is detected by the Hough transform. Finally, the area of the connected region is determined for the image processed by the ViBe algorithm to remove the tiny noise in detection images. The simulation results show that the proposed algorithm can not only detect foreign substances in transparent liquid but also detect foreign substances in the turbid liquid. © 2018 SPIE and IS&T [DOI: 10.1117/1.JEI.27.4.043015] [ABSTRACT FROM AUTHOR]

The research in the robot vision becomes more and more attractive since the demand for robot is growing. To detect and avoid the obstacles in the outdoor environment is an important task for a moving robot. In this paper, we propose a real-time detection of the obstacle based on the computer vision with single camera. The dense optical flow method is adopted to extract the training data for a classifier model using support vector machine (SVM). The speeded-up robust features method is used to detect the interest points to be verified as the obstacle points or not by a SVM classifier. Moreover, a measurement of the spatial weighted saliency map is proposed to highlight the pixels of the obstacle. Finally, the obstacle points and the saliency map are combined to locate the region of the obstacle. The experimental results show that the proposed algorithm can effectively detect the obstacle in the outdoor environment. [ABSTRACT FROM AUTHOR]

Open-air optical flow estimation is crucial for autonomous driving and traffic monitoring. However, it is still challenging to obtain high-precision optical flow under severe imaging environments. To alleviate this problem, we propose a self-adaptive optical flow estimation framework for harsh outdoor scenes, which consists of a frame-adaptive learnable module (FAL-module), a streak-invariant feature extractor (SFE), and a feature matching and flow propagation network. In the framework, three components are jointly learned in an end-to-end manner. The input frames can be adaptively enhanced without sacrificing optical flow-related information by FAL-module. And the SFE decreases the influence of streak-like noise generated by rainy and snowy. Compared with the state-of-the-art methods, the experimental results show that our approach can significantly improve the optical flow estimation quality in multiple harsh outdoor scenes without performance penalties in normal scenes. [ABSTRACT FROM AUTHOR]

• A machine vision method was developed to monitor fan operations in animal houses. • Aggregability and discriminability were found for the displacement matrix of pixels. • Proper resolution and threshold were recommended in using the proposed method. • Fan running state was 100% identified with the error of 1.84% for airflow rate. Real-time monitoring of fan operation is essential for supervising and regulating the airflow rate in a mechanical ventilated animal house. This study proposed a real-time method based on image processing and mathematical modelling to monitor the running state and airflow rate of exhaust fans to ensure the adequate ventilation in a livestock house. Videos and actual airflow rates of a running fan were collected at different operating levels (15, 25, 35, and 45 Hz). The Hough transformation was used to locate the fan in the image, and the dense optical flow was used to calculate the displacement of pixels during fan operation. It was found that the vector sum of the displacements for all pixels in the image of a running fan tends to be zero and the variances of the displacements were discriminable at different operation levels, which can be potentially used to monitor the running state and airflow rate of an exhaust fan. Running state (yes or no) of the fan was identified according to the proportion of moving pixels in the target area and its airflow rate can be estimated by curve fitting between the variance of pixel displacement and measured airflow rates at different operating levels. Buffer time window was used for smoothing to increase the accuracy and stability of fan operation monitoring. Using the proposed method, running state of the fan could be 100 % identified and all fans in sight of the camera could be monitored at the same time. In estimating airflow rate, average absolute percentage error of the propose method was 1.84 % (115.30 m3/h) and it reaches the detection limitation when the operating levels was above 72 % of the maximum airflow rate (50 Hz, 18,000 m3/h) due to the time gap between fan running and the frame of imaging. Furthermore, results showed that a minimum resolution of 1280 × 720 was required when using this method to estimate airflow rate of a fan and the higher resolution of images could improve the estimation accuracy. [ABSTRACT FROM AUTHOR]

It is known that to get the contours and segmentations of moving cars is the essential step of image processing in intelligent traffic tracking systems. As an effective way, the optical flow algorithm is widely used for this kind of applications. But in traditional gradient-based approaches, in order to make the data responding to the edges of moving objects expand to the area, which gray level is flat, it needs to keep the iteration times large enough. It takes a large amount of calculation time, and the accuracy of the result is not as good as expected. In order to improve the numerical reliability of image gradient data, Hessian matrix distinguishing, Gaussian filtering standard deviation amending, mean model amending and multi-image comparing, the four algorithms were investigated by applying them to track moving objects. From the experimental results, it is shown that both the calculation convergence speed and accuracy of our methods have greatly improved comparing with traditional algorithms, the feasibility and validity of those methods were confirmed.

Computer vision techniques are applied to perform automatic wildlife surveying and animal monitoring. Animal detection in aerial videos is challenging because of the complexity of wild environments. In this paper, a method for moving animal detection is proposed by taking advantage of global patterns of pixel motion. In the video dataset, where animals make obvious movement against the background, motion vectors of each pixel are estimated by applying optical flow methods. A coarse segmentation then removes most parts of the background by applying a pixel velocity threshold. Based on the segmented regions, another threshold was employed to filter out negative candidates that could belong to the background. The pros and cons of this method are discussed. [ABSTRACT FROM AUTHOR]

Image mosaicing is a trendy research area in computer vision in recent years. Instead of images, video frames can be mosaiced to form a larger aggregate. In both cases, merging computations generally depend on the common parts in consecutive images, which are called overlapping regions. The presence of mobile devices with a camera provides higher image and video acquisition; hence video mosaicing techniques can be utilized to digitize and to create high resolution historical document images for later use in browsing and retrieval systems. Here, we present a technique to create high resolution images of historical Ottoman documents from video. The video captures the frames while tracing the document in row major ordering. In the preprocessing steps, the frames are oriented automatically by the help of tracking the optical flow, and the keyframes are selected such that a sufficient overlapping region in between is guaranteed. Then the scanning phase traces these keyframes in horizontal and vertical manner to create the final video mosaic image in higher resolution. Both horizontal and vertical scan operations utilize a distance vector to compute similarity between two adjacent images. The effectiveness of the video mosaicing approach for historical Ottoman documents is evaluated, and the results of the experiments show that the approach is expressive yet effective to create high resolution Ottoman images. [ABSTRACT FROM AUTHOR]

Lip event detection is of crucial importance to the better understanding of visual speech perceptually between humans and computers. In this paper, we address an efficient lip event detection approach using oriented histograms of regional optical flow (OH-ROF) and low rank affinity pursuit. First, we align the extracted lip region sequences to reduce the impact of irrelevant motion caused by the moving cameras. Then, an optical flow field is calculated from these sequentially stabilized images and an efficient descriptor, namely OH-ROF, is presented to discriminatively code the visual appearance of each motion frame, whereby each lip motion clip can be represented by a sequence of OH-ROF vectors as its signature. Subsequently, we detect the visual silence event based on the small flow magnitude, and further propose a low rank affinity pursuit method to determine the visual speech event that incorporates the lip-dynamic states of mouth opening and closing. As a result, various kind of lip motion events can be appropriately estimated. The proposed approach neither requires any training set on the labeled videos nor learns the lip motion priors of each visual event in an unconstrained video. Experiments show a promising result in comparison with the state-of-the-art counterparts. [ABSTRACT FROM AUTHOR]

The high-level video analytics PC subsystem as PCIe add-in card is developed for surveillance, retail marketing tool, and various applications which require acceleration of computer vision analytics for real-time video. The PC subsystem has multicore architecture SoC, which comprises proposed computer vision engine (CVE) based on the filter, optical flow extraction, object detection accelerators, and memory sharing using the L3 cache. The optical flow accelerator has parallel and pipelined datapath with tight-coupled DMA control scheme to improve performance and less latency. The object detection accelerator has the capability of online learning from images so that it can give a robustness of tracking targets. L3 cache contributes to shortening memory access latency. Heterogeneous structure with the CPU and CVE accelerator achieves performance at 3500 GOPS, and power efficiency at 1871 GOPS/W, it is approximately 2–15$\times$ higher performance and higher efficiency than other SoCs. [ABSTRACT FROM PUBLISHER]

Human facial expressions detection plays a central role in pervasive health care and it is an active research field in computer vision. In this paper, a novel method for facial expression detection from dynamic facial images is proposed, which includes two stages of feature extraction and facial expression detection. Firstly, Active Shape Model (ASM) is used to extract the local texture feature, and optical flow technique is determined facial velocity information, which is used to characterize facial expression. Then, fusing the local texture feature and facial velocity information get the hybrid characteristics using Bag of Words. Finally, Multi-Instance Boosting model is used to recognize facial expression from video sequences. In order to be learned quickly and complete the detection, the class label information is used for the learning of the Multi-Instance Boosting model. Experiments were performed on a facial expression dataset built by ourselves and on the JAFFE database to evaluate the proposed method. The proposed method shows substantially higher accuracy at facial expression detection than has been previously achieved and gets a detection accuracy of 95.3%, which validates its effectiveness and meets the requirements of stable, reliable, high precision and anti-interference ability etc. [ABSTRACT FROM AUTHOR]