Search

Simultaneous and proportional control (SPC) methods based on surface electromyogram (sEMG) can provide a more intuitive and natural interaction in rehabilitation assistive robots and prostheses. Recently, an increasing number of researchers have utilized deep learning methods to continuously estimate movements across multiple degrees of freedom (DoFs). It stands for higher dimensional and more complex target space, which requires more comprehensive feature extraction methods. However, the existing feature extraction methods for deep learning approaches can hardly provide adequate information for multiple DoFs complex movement estimation. Therefore, we proposed a novel sEMG feature extraction method for deep learning models to more accurately predict the joint angles simultaneously on ten DoFs and tested it on eight complex movements. The experiment was carried out on four commonly adopted deep learning methods to compare the performance of the proposed feature set with others. For the four deep learning methods, the average Pearson correlation coefficient (CC) value was around 0.85 and 0.61 for the proposed feature set and raw sEMG data, respectively. Besides, the feature extraction only takes 6 μs per 100 ms analysis window. This study suggested that the proposed feature set would improve deep learning method's application prospect on SPC for rehabilitation robotic systems.

Recovering human pose from RGB images and videos has drawn increasing attention in recent years owing to minimum sensor requirements and applicability in diverse fields such as human-computer interaction, robotics, video analytics, and augmented reality. Although a large amount of work has been devoted to this field, 3D human pose estimation based on monocular images or videos remains a very challenging task due to a variety of difficulties such as depth ambiguities, occlusion, background clutters, and lack of training data. In this survey, we summarize recent advances in monocular 3D human pose estimation. We provide a general taxonomy to cover existing approaches and analyze their capabilities and limitations. We also present a summary of extensively used datasets and metrics, and provide a quantitative comparison of some representative methods. Finally, we conclude with a discussion on realistic challenges and open problems for future research directions.

With the increasing importance of gaze tracking technology in the detection of driver's attention and fatigue, the estimation of the driver's gaze zone has become a key point and a hot issue for safe driving. Although there has been significant improvement in the research of the driver's gaze area at present, a general gaze zone estimation system for different subjects and perspectives is still lacking. In order to better accomplish this task, this paper uses the “Columbia Gaze Data Set” that contains 56 subjects with different gaze directions and head poses to estimate the driver's gaze zone. After pre-processing the pictures such as normalization and data augmentation, the image dataset is modeled and predicted using the improved VGG16 convolutional neural network structure constructed by Keras, and the 5 head poses and 21 gaze points of each head pose in the dataset are reclassified and divided into 8 gaze areas, corresponding to the gaze zone of the driver.

The low-cost depth cameras have facilitated the research of human action recognition in the last decades. Despite various approaches have been presented to improve the recognition accuracy, they are rarely extended to online recognition task in clutter scenes. In this paper, we propose an effective approach, which is insensitive to various temporal duration and adequate for complex background, for human action recognition using depth sequences. By embedding the skeleton information into depth maps, the human body is partitioned to a set of motion parts, which could take account of the geometrical structure of human body and contribute to the recognition task in complex background. A local spatio-temporal scaled pyramid is applied to obtain compact local feature representation. The simplified Fisher vector encoding method is introduced to aggregate local coarse features into a discriminative representation with unified form. The proposed approach is validated on three public benchmark datasets, i.e., MSR Daily Activity 3D, MSR Action Pairs, and MSR Action 3D. The experimental results demonstrate the effectiveness and feasibility of proposed approach for real-time applications.

Rainstorms, insect swarms, and galloping horses produce “sound textures,”which are the resulting natural sounds of many similar acoustic events. With new achievements emerging regularly for generative models, the deep convolutional neural network (CNN) has proven to be a tremendously successful approach for image and sound synthesis. Existing state-of-the-art sound texture generative models simply treat sound texture signals as 1-D images while discarding the difference between the human vision and auditory systems. This paper considers mel-frequency statistical features, which are designed according to the human auditory system and have been viewed as the dominant features for sound identification. We first construct a CNN structure for extracting mel-frequency features from sounds losslessly. This structure is called mel-frequency CNN (MF-CNN). Next, we investigate a novel sound texture generative model by incorporating the MF-CNN into a convolutional generative network composed of cascading upsampling groups. A jointly alternating back propagation algorithm is proposed to train the overall network. The feedback of the MF-CNN is used to advise the gradients in the inferential and learning back propagation to make the mel-frequency features of the synthesized sounds more similar to the natural ones. Moreover, the proposed generative model can also be extended to other sound synthesis tasks.

Human action recognition based on 3D data is attracting increasing attention because it could provide more abundant spatial and temporal information compared with RGB videos. The challenge of the depth map based method is to capture the cues between spatial appearances and temporal motions. In this paper, we propose a straightforward and efficient framework for modeling the human action based on depth map sequences, considering the short-term and long-term dependencies. A frame-level feature, termed depth-oriented gradient vector (DOGV), is developed to capture the appearance and motion in a short-term duration. For a long-term dependence, we construct convolutional neural networks (CNNs) based backbone to aggregate frame-level features in the space and time sequence. The proposed method is comprehensively evaluated on four public benchmark datasets, including NTU RGB+D, NTU RGB+D 120, PKU-MMD and UOW LSC. The experimental results demonstrate that the proposed approach can solve the problem of 3D human action recognition in an efficient way and achieve the state-of-the-art performance.

Depth sequences are useful for action recognition since they are insensitive to illumination variation and provide geometric information. Many current action recognition methods are limited by being computationally expensive and requiring large-scale training data. Here we propose an effective method for human action recognition using depth sequences captured by depth cameras. A multi-resolution operation, the spatial Laplacian and temporal energy pyramid (SLTEP), decomposes the depth sequences into certain frequency bands in different space and time positions. A spatial aggregating and fusion scheme is applied to cluster the low-level features and concatenate two different feature types extracted from low and high frequency levels, respectively. We evaluate our approach on five public benchmark datasets (MSRAction3D, MSRGesture3D, MSRActionPairs, MSRDailyActivity3D, and NTU RGB+D) and demonstrate its advantages over existing methods and is likely to be highly useful for online applications.

Since FCN [1] added deep neural network to image segmentation technology, the image segmentation effect has been significantly improved in recent years. For example, the VOC2012 data set has been improved from 40% to more than 80%, and the effect has doubled. . However, the improvement of the effect is only for the training set and the verification set of the same data set. When the crossdata set is verified, the accuracy rate will drop sharply. In response to this problem, we propose to apply the CycleGAN [2] network to image segmentation to learn the style features between different datasets. It turns out that the processing can improve the effect of image segmentation between different datasets.

This paper aims to highlight vision related tasks centered on “shrimps”. With the further study of computer vision of marine life, we show that “shrimps” has been largely neglected in comparison to other objects. In image classification, the degree of visual separation between different shrimp categories is highly uneven, the appearance of some categories in same genus is very similar, and it is more difficult to distinguish than others. Based on the classification model of traditional convolutional neural network, this paper presents a method of merging shallow and deep features extracting feature maps from different levels according to the characteristics of shrimp. In order to facilitate future shrimps-related research, we present our on-going effort in collecting a dataset in this paper, “ShrimpX”, that covers not only shrimps and lobsters living in the sea, but also some freshwater shrimps. The “ShrimpX” dataset contains a variety of shrimp images crawled from image search engines. Experimental results on the “ShrimpX” dataset demonstrate that the proposed method can effectively improve the accuracy.

Due to the advantages of retrieval speed and storage space, deep hashing methods have become a research hotspot in the field of large-scale image retrieval. Most of existing deep hashing methods pay close attention to similarity between images without images at the top of the ranking list similar to query targets. In the paper, a novel deep hashing model is proposed to preserve top images similar to the query images and optimize the quality of hash codes for image retrieval. Specifically, the optimized AlexNet is utilized to extract discriminative image representations and learn hashing functions simultaneously. The loss function based on acceleration strategy is designed to ensure similarity between returned images at the top of the ranking list and query images. In addition, we implement the model training in a batch-process fashion to low the image storage. Moreover, our extensive experiments on standard benchmarks demonstrate that our method outperforms several state-of-the-art deep hashing methods.

In highly sophisticated network attacks, command-and-control (C&C) servers always use domain generation algorithms (DGAs) to dynamically produce several candidate domains instead of static hard-coded lists of IP addresses or domain names. Distinguishing the domains generated by DGAs from the legitimate ones is critical for finding out the existence of malware or further locating the hidden attackers. The word-based DGAs disclosed in recent network attack events have shown significantly stronger stealthiness when compared with traditional character-based DGAs. In word-based DGAs, two or more words are randomly chosen from one or more specific dictionaries to form a dynamic domain, these regularly generated domains aim to mimic the characteristics of a legitimate domain. Existing DGA detection schemes, including the state-of-the-art one based on deep learning, still cannot find out these domains accurately while maintaining an acceptable false alarm rate. In this study, we exploit the inter-word and inter-domain correlations using semantic analysis approaches, word embedding and the part-of-speech are taken into consideration. Next, we propose a detection framework for word-based DGAs by incorporating the frequency distribution of the words and that of part-of-speech into the design of the feature set. Using an ensemble classifier constructed from Naive Bayes, Extra-Trees, and Logistic Regression, we benchmark the proposed scheme with malicious and legitimate domain samples extracted from public datasets. The experimental results show that the proposed scheme can achieve significantly higher detection accuracy for word-based DGAs when compared with three state-of-the-art DGA detection schemes.

Human action recognition is a hot research topic in computer vision, which has extensive applications including human-computer interaction, robot and surveillance. Some outstanding human action datasets have been published. However, they still have some defects, such as the limited modality types, camera views, complexity of scene and action categories. Furthermore, most of them are devised for a subset of the action recognition, such as single-modal recognition and single-view recognition. In this paper, we propose a multi-modal cross-view human action dataset (named CAS-YNU MCHAD). It consists of five synchronous data modalities, including RGB image, depth maps, human segment maps, skeleton data and inertial data. Our dataset contains 14782 action samples from 50 subjects and 10 actions in two perspectives. Extensive benchmark experiments on this dataset are conducted with the state-of-the-art recognition approaches. Experimental results illustrate that it is challenging on cross-view action recognition. We also provide the baseline for the assessment of existing recognition methods.

Human action recognition has been an active research topic for its wide applications. Most researches focus on the recognition based on one single modality sensor. In this paper, we present a novel approach for human action recognition, which is based on feature-level fusion of depth and inertial sensor. We extract Fast Fourier Transform (FFT) coefficients from acceleration signals and Histograms of Oriented Gradients (HOG) features from Motion Response Maps (MRM). After obtaining these two modality feature vectors, we adopt Discriminant Correlation Analysis (DCA) to learn a fused feature descriptor with better discriminating ability. To evaluate the effectiveness and efficiency of the proposed approach, we conduct experiments on the multimodal human action database CAS-YNU-MHAD. Experimental results demonstrate the fused feature descriptor exhibits a strong and stable performance in improving the recognition accuracy. Moreover, our approach has a low computational complexity and can be employed in real-time systems.

A new method is proposed to calculate the background in video sequences. The optical flow is estimated to determine the local regions occupied by moving objects. The background image is calculated by an efficient averaging process excluding the moving object regions, which overcomes the foreground-occluding problem in direct averaging method for background estimation. The experiments for traffic video processing prove the method’s effectiveness and robustness.

A physics-inspired deformable grid model is proposed for image structure representation. The attracting force between adjacent points is defined according to the gray-scale difference, which causes the grid to deform. The final grid shape after deformation can represent image structure, based on which a segmentation method is proposed. The experiments indicate the method’s effectiveness.

Visual SLAM is widely used in robotics and computer vision. Although there have been many excellent achievements over the past few decades, there are still some challenges. 2D feature-based SLAM algorithm has been suffering from the inaccurate or insufficient correspondences while dealing with the case of textureless or frequently repeating regions. Furthermore, most of the SLAM systems cannot be used for long-term localization in a wide range of environment because of the heavy burden of calculating and memory. In this paper, we propose a robust RGB-D keyframe-based SLAM algorithm. The novelty of proposed approach lies in using both 2D and 3D features for tracking, pose estimation and bundle adjustment. By using 2D and 3D features, the SLAM system can achieve high accuracy and robustness in some challenging environments. The experimental results on TUM RGB-D dataset [1] and ICL-NUIM dataset [2] verify the effectiveness of our algorithm.

A novel vector field transform is proposed for digital images, which is inspired by the electro-static analogy. By introducing the factor of gray-scale difference into the repulsion vector between image points, the diffusing vector field is obtained by the transform. A new image segmentation method is then presented based on the diffusing vector field. The experimental results have proved the method’s effectiveness.

In this paper, we present an effective method to recognize human actions from sequences of depth maps, which are captured by a consume depth sensor. In our approach, we first project each frame of a depth sequence onto three orthogonal planes and generate the depth motion sequence (DMS) between two consecutive frames from the three projected views. Then we propose a spatio-temporal cuboid pyramid (STCP) to subdivide the DMS volumes into a set of spatial cuboids on scaled temporal levels. And a cuboid fusion scheme is presented to concatenate the histograms of oriented gradients (HOG) features extracted from the spatial cuboid. The proposed approach is evaluated on three public benchmark datasets, i.e., MSRAction3D, MSRGesture3D and MSRActionPairs dataset. The experimental results demonstrate that the proposed method achieves state-of-the-art performance.

The mathematical model is established for a vision system including a camera and a multi-linear structured light sensor. In the model, a novel calibration method is proposed to calibrate multi-linear structured light using a coplanar reference target. In order to calibrate all structured light planes, firstly the two of those should be calibrated according to points on the coplanar reference target in different positions. In the model, the camera coordinate is considered as a global world coordinate, and all local world coordinates are transformed to the camera coordinate, so that all points can be reconstructed in the camera coordinate. In this way, when a camera and a light sensor are fixed together as a vision system, the whole system can be moved freely and easily, which is useful in engineering applications. In addition, a new algorithm to calculate cross ratio is presented, which decreases amount of calculation and improves the calculation precision effectively. Compared to similar technologies, the model decreases the cost of calibration equipment, simplifies the calibration procedure and improves the calculation precision. Experimental results show its convenience and effectiveness.

With the promptly growing population of the low-cost Microsoft Kinect sensor, action recognition, which is a hard yet important problem in computer vision, has been received substantial attention. However, most existing approaches in action recognition spend much time on feature detection even though these methods can achieve high recognition rates. In this paper, we propose a local mean spatio-temporal feature (LMSF) to speed up depth image based action recognition. In particular, we solve the problem from three aspects: (1) associate the 4D normals by a local mean spatio-temporal neighborhood; (2) extract motion frames by detecting the differences between consecutive frames; (3) reduce redundant normals extracted from depth cloud points by sparse coding. The proposed approach is tested on two public benchmark datasets, i.e., MSRAction3D and MSRGesture3D. Experimental results demonstrate the advantages of our improvement method and the state-of-the-art performance on processing speed.

Moving object detection is one of the key technologies for intelligent video monitoring systems. For real-time detection of moving object in the surveillance scene, the general and simple method is based on background image difference. However, it requires the accurate current background image and the approach for automatic background updating along with the illumination variance is difficult to design and implement. This limits its applications. To solve the above problem, a new self-adaptive background approximating and updating algorithm based on optical flow theory is presented for the traffic surveillance scene in this paper. To detect the moving regions of interest in the scene, the difference image between the current frame and the updating background is first obtained by using a color image difference model, and then a self-adaptive thresholding segmentation method for moving object detection based on the Gaussian model is developed and implemented. Moreover, an effective shadow-eliminating algorithm based on contour information and color features is developed. Experimental results demonstrate that the proposed background updating method can update the background exactly and rapidly along with the variance of illumination, the self-adaptive thresholding segmentation method based on the Gaussian model can extract the moving object regions accurately and completely, and the shadow can be eliminated accurately. This is the foundation for further objects recognition and understanding.

In order to improve the precision of 3D reconstruction, a new composite coding way based on structure light is proposed. In this paper, color stripe pattern is used as basic coding element and intensity properties are introduced for every color stripes to design coding pattern. This coding pattern can be used to extract stripe peak and edge information on sub-pixel level. In the reconstruction we use both peak-based reconstruction and edge-based reconstruction. Experimental results show that the combination of two kinds of reconstruction style can ensure higher reconstruction accuracy.

In this paper, we propose a new framework in pedestrian detection using a two-step classification algorithm, which is a “coarse to fine” course. The framework consists of a full-body detection (FBD) step and a head-shoulder detection (HSD) step. The FBD step uses fusion of Haar-like and HOG features to get better performance, and the HSD step utilizes edgelet features for classification and detection. The pedestrian data is obtained from MIT, INRIA dataset and surveillance videos for training. The experiment carried out on videos from campus and CAVIAR dataset illustrates that the proposed method is robust and feasible enough for pedestrian detection and could handle occlusions more accurately than other methods.

This paper presents a color structured light technique for recovering object shape from images. Coding strategy is one of the most important techniques because it effects the measure precision and efficiency directly. Unlike traditional encoding methods, we design new coloured stripe patterns in order to locate both intensity peaks and edges. Then edge detection method with sub-pixel accuracy is performed and the detected stripes centres and the edges between adjacent stripes are matched with the projected pattern. The experimental results show that our methods are very powerful in terms of accuracy and resolution.

This paper is concerned with shadow removal in color images. We propose a combined algorithm which is based on colorimetry and retinex theory. Our method can effectively remove shadow in color images, and meanwhile avoid intrinsic disadvantages of the two algorithms if they are used individually. It can be used as a pre-process step in some computer vision tasks such as image segmentation, motion tracking and objective recognition.