Search

Extensive studies have been conducted on human action recognition, whereas relatively few methods have been proposed for hand action recognition. Although it is very natural and straightforward to apply a human action recognition method to hand action recognition, this approach cannot always lead to state-of-the-art performance. One of the important reasons is that both the between-class difference and the within-class difference in hand actions are much smaller than those in human actions. In this paper, we study first-person hand action recognition from RGB-D sequences. To explore whether pretrained networks substantially influence accuracy, 8 classic pretrained networks and one pretrained network designed by us are introduced for extracting RGB-D features. A Lie group is introduced for hand pose representation. Ablation studies are conducted to compare the discriminative power of the RGB modality, depth modality, pose modality, and their combinations. In our method, a fixed number of frames are randomly sampled to represent an action. This temporal modeling strategy is simple but is proven more effective than both the graph convolutional network (GCN) and the recurrent neural network (RNN), which are widely adopted by conventional methods. Evaluation experiments on two public datasets demonstrate that our method markedly outperforms recent baselines.

In this article, a novel method for simultaneous localization and mapping (SLAM) named surface elements (surfel) point SLAM (SP-SLAM) is proposed using an RGB-D camera. The key idea of SP-SLAM is to use not only keypoints but also surfels as features to cope with both high texture and low texture environments. By decomposing a surface into a small number of surfels, the method can represent spacious environments using a relatively small amount of memory. To optimize the poses of points, surfels, and cameras altogether, new objective functions are proposed, and a new bundle adjustment using these objective functions is developed. The proposed SP-SLAM runs in real time on a central processing unit as in other feature-based visual SLAM methods but works better than them not only in high texture but also in low texture environments, overcoming well-known drawbacks of feature-based visual SLAM with degradation in low texture environments. The proposed method is applied to benchmark datasets and its effectiveness is demonstrated by comparing against those of previous methods in terms of localization accuracy.

Agricultural crop production is a major contributing element to any country’s economy. To maintain the economic growth of any country plants disease detection is a leading factor in agriculture. The contribution of the proposed algorithm is to optimize the extracted information from the available resources for the betterment of the result without any additional complexity. The proposed technique basically localizes the leaf region prior to the image classification into healthy and diseased. The novelty of this work is to fuse the information extracted from the available resources and optimize it to enhance the expected outcome. The leaf colors are analyzed using color transformation for the seed region identification. The mapping of a low-dimensional RGB color image into L*a*b color space provides an expansion of the spectral range. The neighboring pixels-based leaf region growing is applied on the initial seeds. In order to refine the leaf boundary and the disease-affected areas, we employed a random sample consensus (RANSAC) for suitable curve fitting. The feature sets using bag of visual words, Fisher vectors, and handcrafted features are extracted followed by classification using logistic regression, multilayer perceptron model, and support vector machine. The performance of the proposal is analyzed through PlantVillage datasets of apple, bell pepper, cherry, corn, grape, potato, and tomato. The simulation-based analysis of the proposed contextualization-based image categorization process outperforms as compared with the state of arts. The proposed approach provides average accuracy and area under the curve of 0.932 and 0.903, respectively.

Previous neural network accelerators tailored to video analysis only accept data of RGB/YUV domain, requiring decompressing the video that are often compressed before transmitted from the edge sensors. A compressed video processing accelerator can alleviate the decoding overhead, and gain performance speedup by operating on more compact input data. This work proposes a novel deep learning accelerator architecture, Alchemist, which is able to predict results directly from the compressed video bitstream instead of reconstructing the full RGB images. By utilizing the metadata of motion vector and critical blocks extracted from bitstreams, Alchemist contributes to remarkable performance speedup of 5x with negligible accuracy loss. Nevertheless, we still find that the original compressed video coded by standard algorithms such as H.264 is not suitable to be directly manipulated, due to diverse compressed structures. Although obviating the requirement to recover all RGB frames, the accelerator must parse the entire compressed video bitstream to locate reference frames and extract useful metadata. If we combine the video codec with the proposed compressed video analysis, additional optimizations can be obtained. Therefore, to cope with the mismatch between current video coding algorithms such as H.264 and neural network-based video analysis, we propose a specialized coding strategy to generate compressed video bitstreams more suitable for transmission and analysis, which further simplifies the decoding stage of video analysis and is capable of achieving significant storage reduction.

The head-mounted displays (HMDs) provide immersive experiences in virtual reality (VR). However, the face interactions are limited due to the serious occlusion of the user’s face. Existed approaches try to recover the user’s facial expression by adding additional sensors to HMD. In this paper, we develop a novel framework to reconstruct the userb’s 3D face in VR only using an RGB camera. Given a reference face, a realistic full-textured avatar is created by fitting a 3D Morphable Model (3DMM). A self-supervised UV map Generative Adversarial Network (GAN) is proposed to make the facial texture look more realistic. Next, we propose a novel landmark detection method to locate the landmark positions under HMD occlusion since facial landmarks are commonly used for driving the avatar. To this end, we synthesize a face dataset with HMD. Our method is easy to build and popularize with low cost. The experiments on the synthetic and real HMD data demonstrate that the proposed method can detect landmark accurately and restore facial expressions faithfully despite the large occlusion of HMD.

Pesticide residues have long been a significant aspect of food safety, which has always been a major social concern. This study presents research and analysis on the identification of pesticide residue fast detection cards based on the enzyme inhibition approach. In this study, image recognition technology is used to extract the color information RGB eigenvalues from the detection results of the quick detection card, and four regression models are established to quantitatively predict the pesticide residue concentration indicated by the quick detection card using RGB eigenvalues. The four regression models are linear regression model, quadratic polynomial regression model, exponential regression model and RBF neural network model. Through study and comparison, it has been shown that the exponential regression model is superior at predicting the pesticide residue concentration indicated by the rapid detection card. The correlation value is 0.900, and the root mean square error is 0.106. There will be no negative prediction value when the expected concentration is near to 0. This gives a novel concept and data support for the development of image recognition equipment for pesticide residue fast detection cards based on the enzyme inhibition approach.

Most existing literature has ignored the potential effects that color intensity may have on art prices (bearing a few recent exceptions). We examine 1,627 paintings executed by the “Big five” Latin American artists (Rivera, Tamayo, Lam, Matta, and Botero) and sold at Sotheby’s and Christie’s between 2003 and 2017 to analyze this impact. We find strong evidence indicating that paintings that are more intense in color fetch higher prices, but only up to a certain degree (paintings whose color is “too intense”, “too vivid” or “too dark” actually fetch lower prices). To the best of our knowledge, these results are the first to confirm, for the case of the art market, early experimental evidence in the psychology literature pointing to the existence of an inverse “U” pattern on the preferences for color intensity. Our findings have implications for other areas such as psychology and consumer behavior.

Driver behavior recognition has become one of the most important tasks for intelligent vehicles. This task, however, is very challenging since the background contents in real-world driving scenarios are often very complex. More critically, the difference between driving behaviors is often very minor, making it extremely difficult to distinguish them. Existing methods often rely only on RGB frames (or skeleton data), which may fail to capture the minor differences between behaviors and appearance information of objects simultaneously and thus fail to achieve promising performance. To address the above issues, in this paper, we propose a bidirectional posture-appearance interaction network (BPAI-Net), which simultaneously considers RGB frames and skeleton (\ie, posture) data for driver behavior recognition. Specifically, we propose a posture-guided convolutional neural network (PG-CNN) and an appearance-guided graph convolutional network (AG-GCN) to extract appearance and posture features, respectively. To exploit the complementary information between appearance and posture, we use the appearance features from PG-CNN for guiding AG-GCN to exploit the contextual information (e.g., nearby objects) to enhance posture features. Then, we use the enhanced posture features from AG-GCN to help PG-CNN focus on critical local areas of video frames that are related to driver behaviors. In this sense, we are able to use the interaction between two modalities to extract more discriminative features and thus improve the recognition accuracy. Experimental results on Drive&Act dataset show that our method outperforms state-of-the-art methods by a large margin (67.83% vs. 63.64%). Furthermore, we collect a bus driver behavior recognition dataset and yield consistent performance gain against baseline methods, demonstrating the effectiveness of our method in real-world applications. The source code and trained models are available at github.com/SCUT-AILab/BPAI-Net/.

Thermal sensors play an important role in intelligent transportation system. This paper studies the problem of RGB and thermal (RGBT) tracking in challenging situations by leveraging multimodal data. A RGBT object tracking method is proposed in correlation filter tracking framework based on short term historical information. Given the initial object bounding box, hierarchical convolutional neural network (CNN) is employed to extract features. The target is tracked for RGB and thermal modalities separately. Then the backward tracking is implemented in the two modalities. The difference between each pair is computed, which is an indicator of the tracking quality in each modality. Considering the temporal continuity of sequence frames, we also incorporate the history data into the weights computation to achieve a robust fusion of different source data. Experiments on three RGBT datasets show the proposed method achieves comparable results to state-of-the-art methods.

Head pose estimation is an important problem as it facilitates tasks such as gaze estimation and attention modeling. In the automotive context, head pose provides crucial information about the driver's mental state, including drowsiness, distraction and attention. It can also be used for interaction with in-vehicle infotainment systems. While computer vision algorithms using RGB cameras are reliable in controlled environments, head pose estimation is a challenging problem in the car due to sudden illumination changes, occlusions and large head rotations that are common in a vehicle. These issues can be partially alleviated by using depth cameras. Head rotation trajectories are continuous with important temporal dependencies. Our study leverages this observation, proposing a novel temporal deep learning model for head pose estimation from point cloud. The approach extracts discriminative feature representation directly from point cloud data, leveraging the 3D spatial structure of the face. The frame-based representations are then combined with bidirectional long short term memory (BLSTM) layers. We train this model on the newly collected multimodal driver monitoring (MDM) dataset, achieving better results compared to non-temporal algorithms using point cloud data, and state-of-the-art models using RGB images. We further show quantitatively and qualitatively that incorporating temporal information provides large improvements not only in accuracy, but also in the smoothness of the predictions.

Modern vehicles are equipped with various driver-assistance systems, including automatic lane keeping, which prevents unintended lane departures. Traditional lane detection methods incorporate handcrafted or deep learning-based features followed by postprocessing techniques for lane extraction using frame-based RGB cameras. The utilization of frame-based RGB cameras for lane detection tasks is prone to illumination variations, sun glare, and motion blur, which limits the performance of lane detection methods. Incorporating an event camera for lane detection tasks in the perception stack of autonomous driving is one of the most promising solutions for mitigating challenges encountered by frame-based RGB cameras. The main contribution of this work is the design of the lane marking detection model, which employs the dynamic vision sensor. This paper explores the novel application of lane marking detection using an event camera by designing a convolutional encoder followed by the attention-guided decoder. The spatial resolution of the encoded features is retained by a dense atrous spatial pyramid pooling (ASPP) block. The additive attention mechanism in the decoder improves performance for high dimensional input encoded features that promote lane localization and relieve postprocessing computation. The efficacy of the proposed work is evaluated using the DVS dataset for lane extraction (DET). The experimental results show a significant improvement of $5.54\%$ and $5.03\%$ in $F1$ scores in multiclass and binary-class lane marking detection tasks. Additionally, the intersection over union ($IoU$) scores of the proposed method surpass those of the best-performing state-of-the-art method by $6.50\%$ and $9.37\%$ in multiclass and binary-class tasks, respectively.

The use of depth information, acquired by depth sensors, for salient object detection (SOD) is being explored. Despite the remarkable results from recent deep learning approaches for RGB-D SOD, they fail to fully incorporate original and accurate information to express details of RGB-D images in salient objects. Here, we propose an RGB-D SOD model using a three-input multilevel fusion network (TMFNet), which differs from existing methods based on double-stream networks. In addition to RGB input (first input) and depth input (second input), the RGB image and depth map are combined into a four-channel representation (RGBD input) that constitutes the third input to the TMFNet. The RGBD input generates multilevel features that reflect details of the RGB-D image. In addition, the proposed TMFNet aggregates diverse region-based contextual information without discarding RGB and depth features. Thus, we introduce a cross-fusion module, and benefiting from rich low- and high-level information from salient features, feature fusion enables the improvement of localization of salient objects. The proposed TMFNet achieves state-of-the-art performance on six benchmark datasets for SOD.

The edge information plays a key role in the restoration of a depth map. Most conventional methods assume that the color image and depth map are consistent in edge areas. However, complex texture regions in the color image do not match exactly with edges in the depth map. In this paper, firstly, we point out that in most cases the consistency between normal map and depth map is much higher than that between RGB-D pairs. Then we propose a dual normal-depth regularization term to guide the restoration of depth map, which constrains the edge consistency between normal map and depth map back and forth. Moreover, considering the bimodal characteristic of weight distribution that exists in depth discontinuous areas, a reweighted graph Laplacian regularizer is proposed to promote this bimodal characteristic. And this regularization is incorporated into a unified optimization framework to effectively protect the piece-wise smoothness(PWS) characteristics of depth map. By treating depth image as graph signal, the weight between two nodes is adapted according to its content. The proposed method is tested for both noise-free and noisy cases, and is compared against the state-of-the-art methods on both synthesis and real captured datasets. Extensive experimental results demonstrate the superior performance of our method compared with most state-of-the-art works in terms of both objective and subjective quality evaluations. Specifically, our method is more effective on edge areas and more robust to noises.

Existing domain adaptation (DA) methods generally assume that different domains have identical label space, and the training data are only sampled from a single domain. This unrealistic assumption is quite restricted for real-world applications, since it neglects the more practical scenario, where the source domain can contain the categories that are not shared by the target domain, and the training data can be collected from multiple modalities. In this article, we address a more difficult but practical problem, which recognizes RGB images through training on RGB-D data under the label space inequality scenario. There are three challenges in this task: 1) source and target domains are affected by the domain mismatch issue, which results in that the trained models perform imperfectly on the test data; 2) depth images are absent in the target domain (e.g., target images are captured by smartphones), when the source domain contains both the RGB and depth data. It makes the ordinary visual recognition approaches hardly applied to this task; and 3) in the real world, the source and target domains always have different numbers of categories, which would result in a negative transfer bottleneck being more prominent. Toward tackling the above challenges, we formulate a deep model, called visual-depth matching network (VDMN), where two new modules and a matching component can be trained in an end-to-end fashion jointly to identify the common and outlier categories effectively. The significance of VDMN is that it can take advantage of depth information and handle the domain distribution mismatch under label inequality simultaneously. The experimental results reveal that VDMN exceeds the state-of-the-art performance on various DA datasets, especially under the label inequality scenario.

Data driven method has been investigated in fault diagnosis. This kind of methods usually need of a large number of labeled data in order to obtain a good model. However, the unlabeled data dominates the data availability in the real industries, which impedes the successfully application of these methods. To remedy this situation, this research investigates a new semi-supervised fault diagnosis method, and it aims at training an adaptive model by using the small labeled data and large unlabeled data. Firstly, the fault signal is transformed to the RGB format. Then, Convolutional Neural Network (CNN) is trained to generate the fault features combining with the Correlation Alignment (CORAL) loss. Finally, the Transfer Component Analysis (TCA) is adopted to classify the fault features to improve the prediction accuracy. The proposed semi-supervised method is conducted on three bearing datasets. The experiments with different proportions of labeled data are done to validate the proposed CNN with CORAL and TCA. The comparisons between the proposed CNN model with other popular deep learning methods are also presented. The results show that CNN with CORAL and TCA is potential for fault diagnosis, and the prediction accuracies of proposed method outperform other comparisons methods.

Reconstructing an indoor scene and generating a layout/floor plan in 3D or 2D is a widely known problem. Quite a few algorithms have been proposed in the literature recently. However, most of the existing methods either use RGB-D images, thus requiring a depth camera, or depend on panoramic photos with the assumption that there is little to no occlusion in the rooms. In this work, we proposed generation of layout using an RGB image captured using a simple mobile phone camera. We take advantage of Simultaneous Localization and Mapping (SLAM) to assess the 3D transformations required for layout generation. SLAM technology is built-in in recent mobile libraries such as ARCore by Google. Hence, the proposed method is fast and efficient, while giving the user freedom to generate layout by simply taking a few conventional photos, rather than relying on specialized depth hardware or occlusion-free panoramic photos.

Remote Photoplethysmography (remote PPG) enables contactless monitoring of the cardiac rhythm using video cameras. Prior research has shown the feasibility of video-based atrial fibrillation (AF) and/or flutter (Aflutter) detection in some scenarios, but most exclude patient movement. In this work, we investigate the feasibility of detecting these two cardiac arrhythmias in a regular hospital environment using an RGB camera, where patients were not limited in movement during the recording process. Data of 56 patients was collected before and after a scheduled cardioversion treatment. Using the data and machine learning models, we developed three models: First, a model to detect only AF from the data excluding any Aflutter cases. Here we report a sensitivity of 94.5% and a specificity of 89.3% with an AUC of 0.966. Second, a model to classify if a cardiac arrhythmia (AF or Aflutter) is present or not. There we report there a sensitivity of 95.6% and a specificity of 91.2% with an AUC of 0.975. Finally, we develop a multi rhythm model, where we classify the data in AF, Aflutter and sinus rhythm separately. The performance of arrhythmia detection is close to the second model, but we note that the distinction between AF and Aflutter is still a challenge. Here we theorize that remote PPG is more sensitive to noise during Aflutter, which will lead to features in Aflutter which are closer to those of AF. To confirm this, we will extensively review the reason of misclassification of Aflutter as AF in future work.

Skeleton-based human action recognition has received extensive attention due to its efficiency and robustness to complex backgrounds. Though the human skeleton can accurately capture the dynamics of human poses, it fails to recognize human actions induced by the interaction between human and objects, making it is of great importance to further explore the interaction between the human and objects for human action recognition. In this paper, we devise the multi-stream interaction networks (MSIN), to simultaneously explore the dynamics of human skeleton, objects, and the interaction between human and objects. Specifically, apart from the traditional human skeleton stream, 1) the second stream explores the dynamics of object appearance from the objects surrounding the human body joints; and 2) the third stream captures the dynamics of object position in regard to the distance between the object and different human body joints. Experimental results on three popular skeleton-based human action recognition datasets, NTU RGB+D, NTU RGB+D 120, and SYSU, demonstrate the effectiveness of the proposed method, especially for recognizing the human actions with human-object interactions.

RGB salient object detection (SOD) has made great progress. However, the performance of this single-modal salient object detection will be significantly decreased when encountering some challenging scenes, such as low light or darkness. To deal with the above challenges, thermal infrared (T) image is introduced into the salient object detection. This fused modal is called RGB-T salient object detection. To achieve deep mining of the unique characteristics of single modal and the full integration of cross-modality information, a novel Cross-Guided Fusion Network (CGFNet) for RGB-T salient object detection is proposed. Specifically, a Cross-Scale Alternate Guiding Fusion (CSAGF) module is proposed to mine the high-level semantic information and provide global context support. Subsequently, we design a Guidance Fusion Module (GFM) to achieve sufficient cross-modality fusion by using single modal as the main guidance and the other modal as auxiliary. Finally, the Cross-Guided Fusion Module (CGFM) is presented and serves as the main decoding block. And each decoding block is consists of two parts with two modalities information of each being the main guidance, i.e., cross-shared Cross-Level Enhancement (CLE) and Global Auxiliary Enhancement (GAE). The main difference between the two parts is that the GFM using different modalities as the main guide. The comprehensive experimental results prove that our method achieves better performance than the state-of-the-art salient detection methods. The source code has released at: https://github.com/wangjie 0825/CGFNet.git.

Diagnosis of mechanical faults in manufacturing systems is critical for ensuring safety and saving costs. With the development of data transmission and sensor technologies, measuring systems can acquire massive amounts of multi-sensor data. Although Deep-Learning (DL) provides an end-to-end way to address the drawbacks of traditional methods, it is necessary to do deep research on an intelligent fault diagnosis method based on Multi-Sensor Data. In this project, a novel intelligent diagnosis method based on Multi-Sensor Fusion (MSF) and Convolutional Neural Network (CNN) is explored. Firstly, a Multi-Signals-to-RGB-Image conversion method based on Principal Component Analysis (PCA) is applied to fuse multi-signal data into three-channel RGB images. Then, an improved CNN with residual networks is proposed, which can balance the relationship between computational cost and accuracy. Two datasets are used to verify the effectiveness of the proposed method. The results show the proposed method outperforms other DL-based methods in terms of accuracy.

Temporal action detection is a challenging task in video understanding, due to the complexity of the background and rich action content impacting high-quality temporal proposals generation in untrimmed videos. Capsule networks can avoid some limitations of the invariance caused by pooling and inability from convolutional neural networks, which can better understand the temporal relations for temporal action detection. However, because of the extremely computationally expensive procedure, capsule network is difficult to be applied to the task of temporal action detection. To address this issue, this paper proposes a novel U-shaped capsule network framework with a k-Nearest Neighbor (k-NN) mechanism of 3D convolutional dynamic routing, which we named U-BlockConvCaps. Furthermore, we build a Capsules Boundary Network (CapsBoundNet) based on U-BlockConvCaps for dense temporal action proposal generation. Specifically, the first module is one 1D convolutional layer for fusing the two-stream with RGB and optical flow video features. The sampling module further processes the fused features to generate the 2D start-end action proposal feature maps. Then, the multi-scale U-Block convolutional capsule module with 3D convolutional dynamic routing is used to process the proposal feature map. Finally, the feature maps generated from the CapsBoundNet are used to predict starting, ending, action classification, and action regression score maps, which help to capture the boundary and intersection over union features. Our work innovatively improves the dynamic routing algorithm of capsule networks and extends the use of capsule networks to the temporal action detection task for the first time in the literature. The experimental results on benchmarks THUMOS14 show that the performance of CapsBoundNet is obviously beyond the state-of-the-art methods, e.g., the mAP@tIoU=0.3, 0.4, 0.5 on THUMOS14 are improved from 63.6% to 70.0%, 57.8% to 63.1%, 51.3% to 52.9%, respectively. We also got competitive results on the action detection dataset of ActivityNet1.3.

RGB-T salient object detection (SOD) aims at utilizing the complementary cues of RGB and Thermal (T) modalities to detect and segment the common objects. However, on one hand, existing methods simply fuse the features of two modalities without fully considering the characters of RGB and T. On the other hand, the high computational cost of existing methods prevents them from real-world applications (e.g., automatic driving, abnormal detection, person re-ID). To this end, we proposed an efficient encoder-decoder network named Context-guided Stacked Refinement Network (CSRNet). Specifically, we utilize a lightweight backbone and design efficient decoder parts, which greatly reduce the computational cost. To fuse RGB and T modalities, we proposed an efficient Context-guided Cross Modality Fusion (CCMF) module to filter the noise and explore the complementation of two modalities. Besides, Stacked Refinement Network (SRN) progressively refines the features from top to down via the interaction of semantic and spatial information. Extensive experiments show that our method performs favorably against state-of-the-art algorithms on RGB-T SOD task while with small model size (4.6M), few FLOPs (4.2G), and real-time speed (38fps). Our codes is available at: https://github.com/huofushuo/CSRNet.

The use of complementary information, namely depth or thermal information, has shown its benefits to salient object detection (SOD) during recent years. However, the RGB-D or RGB-T SOD problems are currently only solved independently, and most of them directly extract and fuse raw features from backbones. Such methods can be easily restricted by low-quality modality data and redundant cross-modal features. In this work, a unified end-to-end framework is designed to simultaneously analyze RGB-D and RGB-T SOD tasks. Specifically, to effectively tackle multi-modal features, we propose a novel multi-stage and multi-scale fusion network (MMNet), which consists of a crossmodal multi-stage fusion module (CMFM) and a bi-directional multi-scale decoder (BMD). Similar to the visual color stage doctrine in the human visual system (HVS), the proposed CMFM aims to explore important feature representations in feature response stage, and integrate them into cross-modal features in adversarial combination stage. Moreover, the proposed BMD learns the combination of multi-level cross-modal fused features to capture both local and global information of salient objects, and can further boost the multi-modal SOD performance. The proposed unified cross-modality feature analysis framework based on two-stage and multi-scale information fusion can be used for diverse multi-modal SOD tasks. Comprehensive experiments (∼92K image-pairs) demonstrate that the proposed method consistently outperforms the other 21 state-of-the-art methods on nine benchmark datasets. This validates that our proposed method can work well on diverse multi-modal SOD tasks with good generalization and robustness, and provides a good multimodal SOD benchmark. Accepted version This work was supported by Ministry of Science and Technology of China - Science and Technology Innovations 2030 (2019AAA0103501), Natural Science Foundation of China (61801303 and 62031013), Guangdong Basic and Applied Basic Research Foundation (2019A1515012031), and Shenzhen Science and Technology Plan Basic Research Project (JCYJ20190808161805519).

International audience; Several works have focused on Simultaneous Localization and Mapping (SLAM), which is a topic that has been studied for more than a decade to meet the needs of robots to navigate in an unknown environment. SLAM is an essential perception functionality in several applications, especially in robotics and autonomous vehicles. RGB-D cameras are among the sensors commonly used with recent SLAM algorithms. They provide an RGB image and the associated depth map, making it possible to solve scale drift with less complexity and create a dense 3D environment representation. Many RGB-D SLAM algorithms have been studied and evaluated on publicly available datasets without considering sensor specifications or image acquisition modes that could improve or decrease localization accuracy. In this work, we deal with indoor localization, taking into account the sensor specifications. In this context, our contribution is a deep experimental study to highlight the impact of the sensor acquisition modes on the localization accuracy, and a parametric optimization protocol for a precise localization in a given environment. Furthermore, we apply the proposed protocol to optimize a depth-related parameter of the SLAM algorithm. The study is based on a publicly available dataset in an indoor environment with a depth sensor. The reconstruction results’ analysis is founded on the study of different metrics involving translational and rotational errors. These metrics errors are compared with those obtained with a state-of-the-art stereo vision-based SLAM algorithm.

Extracting effective spatial-temporal information is significantly important for video-based action recognition. Recently 3D convolutional neural networks (3D CNNs) that could simultaneously encode spatial and temporal dynamics in videos have made considerable progress in action recognition. However, almost all existing 3D CNN-based methods recognize human actions only using RGB videos. The single modality may limit the performance capacity of 3D networks. In this paper, we extend 3D CNN to depth and pose data besides RGB data to evaluate its capacity for spatiotemporal multimodal learning for video action recognition. We propose a novel multimodal two-stream 3D network framework, which can exploit complementary multimodal information to improve the recognition performance. Specifically, we first construct two discriminative video representations under depth and pose data modalities respectively, referred as depth residual dynamic image sequence (DRDIS) and pose estimation map sequence (PEMS). DRDIS captures spatial-temporal evolution of actions in depth videos by progressively aggregating the local motion information. PEMS eliminates the interference of cluttered backgrounds and describes the spatial configuration of body parts intuitively. The multimodal two-stream 3D CNN deals with two separate data streams to learn spatiotemporal features from DRDIS and PEMS representations. Finally, the classification scores from two streams are fused for action recognition. We conduct extensive experiments on four challenging action recognition datasets. The experimental results verify the effectiveness and superiority of our proposed method.

RGB and thermal infrared (RGBT) image saliency detection is a relatively new direction in the field of computer vision. Combining the advantages of RGB images and T images can significantly improve detection performance. Currently, there are only a few methods to work on RGBT saliency detection, and the number of image samples cannot meet the training requirements for deep learning, so it remains valuable to propose an effective unsupervised method. In this paper, we present an unsupervised RGBT saliency detection method based on multi-graph fusion and learning. Firstly, RGB images and T images are adaptively fused based on boundary information to produce more accurate superpixels. Next, a multi-graph fusion model is proposed to selectively learn useful information from multi-modal images. Finally, we implement the theory of finding good neighbors in the graph affinity and propose different algorithms for two stages of saliency ranking. Experimental results on three RGBT datasets show that the proposed method is effective compared with the state-of-the-art algorithms.

Recent years have witnessed the prevalence of using the Siamese network for RGB-T tracking because of its remarkable success in RGB object tracking. Despite their faster than real-time speeds, existing RGB-T Siamese trackers suffer from low accuracy and poor robustness, compared to other state-of-the-art RGB-T trackers. To address such issues, a new complementarity- and distractor-aware RGB-T tracker based on Siamese network (referred to as SiamCDA) is developed in this paper. To this end, several modules are presented, where the feature pyramid network (FPN) is incorporated into the Siamese network to capture the cross-level information within unimodal features extracted from the RGB or the thermal images. Next, a complementarity-aware multi-modal feature fusion module (CA-MF) is specially designed to capture the cross-modal information between RGB features and thermal features. In the final bounding box selection phase, a distractor-aware region proposal selection module (DAS) further enhances the robustness of our tracker. On top of the technical modules, we also build a large-scale, diverse synthetic RGB-T tracking dataset, containing more than 4831 pairs of synthetic RGB-T videos and 12K synthetic RGB-T images. Extensive experiments on three RGB-T tracking benchmark datasets demonstrate the outstanding performance of our proposed tracker with a tracking speed over 37 frames per second (FPS).

Neuromorphic vision sensors such as the Dynamic and Active-pixel Vision Sensor (DAVIS) using silicon retina are inspired by biological vision, they generate streams of asynchronous events to indicate local log-intensity brightness changes. Their properties of high temporal resolution, low-bandwidth, lightweight computation, and low-latency make them a good fit for many applications of motion perception in the intelligent vehicle. However, as a younger and smaller research field compared to classical computer vision, neuromorphic vision is rarely connected with the intelligent vehicle. For this purpose, we present three novel datasets recorded with DAVIS sensors and depth sensor for the distracted driving research and focus on driver drowsiness detection, driver gaze-zone recognition, and driver hand-gesture recognition. To facilitate the comparison with classical computer vision, we record the RGB, depth and infrared data with a depth sensor simultaneously. The total volume of this dataset has 27360 samples. To unlock the potential of neuromorphic vision on the intelligent vehicle, we utilize three popular event-encoding methods to convert asynchronous event slices to event-frames and adapt state-of-the-art convolutional architectures to extensively evaluate their performances on this dataset. Together with qualitative and quantitative results, this work provides a new database and baseline evaluations named NeuroIV in cross-cutting areas of neuromorphic vision and intelligent vehicle.

International audience; The goal of Mixed Reality (MR) is to achieve a seamless and realistic blending between real and virtual worlds. This requires the estimation of reflectance properties and lighting characteristics of the real scene. One of the main challenges within this task consists in recovering such properties using a single RGB-D camera. In this paper, we introduce a novel framework to recover both the position and color of multiple light sources as well as the specular reflectance of real scene surfaces. This is achieved by detecting and incorporating information from both specular reflections and cast shadows. Our approach is capable of handling any textured surface and considers both static and dynamic light sources. Its effectiveness is demonstrated through a range of applications including visually-consistent mixed reality scenarios (e.g. correct real specularity removal, coherent shadows in terms of shape and intensity) and retexturing where the texture of the scene is altered whereas the incident lighting is preserved.

Prevalent video codec implementations deployed in the field often solely support chrominance sub-sampled video data represented by the YCbCr 4:2:0 format. For certain applications like screen sharing, however, chroma sub-sampling leads to disturbing artifacts, especially for text or graphics with thin lines. It is desirable to reduce these artifacts while maintaining compatibility with all user devices. For this reason, an enhancement layer coding framework for chroma-format scalable coding with specific focus on screen content is proposed in this paper. Based on an analysis of screen content data characteristics, the enhancement layer codec is optimized specifically for this content class, is of low algorithmic complexity, and applicable with any image or video codec for base layer compression in a joint coding system. The system is intentionally not designed as a general purpose lossless YCbCr 4:4:4 coding scheme, but instead chooses to close a quality gap that prevalent video codecs do not address. Experimental analysis reveals an average BD-PSNR gain in the RGB domain of 1.0 dB comparing the proposed two-layer scalable coding approach to single layer compression using base layer coding only. In relation to simulcast coding, an average BDR RGB difference of 8.1% is observed.

Automatic modulation classification (AMC) is one of the major challenges for cognitive radio (CR), which can enhance the spectrum utilization efficiency. In this study, a hybrid signal and image-based deep learning model is designed for AMC in CR. A convolutional neural network (CNN) is applied in both the deep learning models. The signal-based CNN (SBCNN) is designed with the optimal filter size for the prediction accuracy, which is used as a pre-training deep learning network to extract features with size 24×1. The features extracted by SBCNN are converted into heat map images, which showed RGB images in the scale range of -30 to +30. Finally, the images are utilized for training and testing the image-based CNN (IBCNN). The dataset used for the experiment is DeepSig:RADIOML2018.01A, which is the latest version. For the IBCNN, the prediction accuracy is 1.96%, 7.99%, and 4.63% higher at signal-to-noise ratio (SNR) 10 dB, and 3.26%, 6.4%, and 4.13% higher at SNR 0 dB as compared to conventional models: ECNN, SCGNet, and LCNN, respectively.

This study compares the utility of multifrequency SAR and Optical multispectral data for land-cover classification of Mumbai city and its nearby regions with a special focus on water body mapping. The L-band ALOS-2 PALSAR-2, X-band TerraSAR-X, C-band RISAT-1, and Sentinel-2 datasets have been used in this work. This work is done as a retrospective study for the dual-frequency L and S-band NASA-ISRO Synthetic Aperture Radar (NISAR) mission. The ALOS-2 PALSAR-2 data has been pre-processed before the implementation of machine learning algorithms for image segmentation. Multi-looking is performed on ALOS-2 PALSAR-2 data to generate square pixels of size 5.78 m and then target decomposition is applied to generate a false-color composite RGB image. While in the case of TerraSAR-X and RISAT-1 datasets, no multi-looking was performed and direct target decomposition was applied to generate false-color composite RGB images. Similarly, for the optical dataset that has a resolution of 10 m, a true color composite, and a false color composite RGB image are generated. For the comparative study between ALOS-2 PALSAR-2 and Sentinel-2 dataset, the RGB images are divided into smaller chunks of size 500*500 pixels each to create a training and testing dataset. Ten image patches were taken from the large dataset, out of which eight patches were used to train the machine learning models Random Forest (RF), K Nearest Neighbor (KNN), and Support Vector Machine (SVM), and two patches were kept for testing and validation purpose. For training the machine learning models, feature vectors are generated using the Gabor filter, Scharr filter, Gaussian filter, and Median filter. For patch 1, the mIOU for true-color composite based Optical image varies from 0.2323 to 0.2866 with the RF classifier performing the best and the mIOU for false-color composite based Optical image varies from 0.4130 to 0.4941 with the RF classifier performing the best while for ALOS-2 PALSAR-2 data, the mIOU varies from 0.4033 to 0.4663 with the RF classifier outperforming the KNN and the SVM classifiers. For patch 2, the mIOU for true-color composite based Optical data varies from 0.3451 to 0.4517 with KNN performing the best and the mIOU for false-color composite based Optical image varies from 0.5156 to 0.5832 with the RF classifier performing the best while for ALOS-2 PALSAR-2 data, the mIOU varies from 0.4600 to 0.5178 with the RF classifier outperforming the KNN and the SVM classifiers. The gap between the performance of ALOS-2 PALSAR-2 data and Sentinel-2 optical data is observed when the IOU of water class is compared, with IOUw for the true-color composite based optical image at a maximum of 0.2525 and for false-color composite based optical image at a maximum of 0.7366 while for ALOS-2 PALSAR-2 data a maximum IOUw of 0.7948 is achieved. The better performance of SAR data as compared to true-color composite based optical image data is due to the misclassification of ground and water classes into urban and forest in the case of the true-color composite based optical dataset which can be attributed to the high similarity between water and forest classes in the case of true-color composite based optical data whereas both these classes are easily separable in case of SAR data. This issue is however resolved by using the false color composite based optical image dataset for the classification task which performs slightly better than ALOS-2 PALSAR-2 data in the overall classification task. However, the SAR data works best in water body detection as notable from the high IOU for water class in the case of SAR data. In addition to the comparative analysis between Sentinel-2 optical and ALOS-2 PALSAR-2 data, land-cover classification has been performed on X-band TerraSAR-X and C-band RISAT-1 data on a single patch and it has been found that the RF classifier performs the best, recording the mIOU 0.5815 for X-band TerraSAR-X data, mIOU of 0.4031 for the C-band RISAT-1 data, and mIOU of 0.6153 for the L-band ALOS-2 data.

Modelling of material appearance from reflectance measurements has become increasingly prevalent due to the development of novel methodologies in Computer Graphics. In the last few years, some advances have been made in measuring the light-material interactions, by employing goniometers/reflectometers under specific laboratory’s constraints. A wide range of applications benefit from data-driven appearance modelling techniques and material databases to create photorealistic scenarios and physically based simulations. However, important limitations arise from the current material scanning process, mostly related to the high diversity of existing materials in the real-world, the tedious process for material scanning and the spectral characterisation behaviour. Consequently, new approaches are required both for the automatic material acquisition process and for the generation of measured material databases. In this study, a novel approach for material appearance acquisition using hyperspectral data is proposed. A dense 3D point cloud filled with spectral data was generated from the images obtained by an unmanned aerial vehicle (UAV) equipped with an RGB camera and a hyperspectral sensor. The observed hyperspectral signatures were used to recognise natural and artificial materials in the 3D point cloud according to spectral similarity. Then, a parametrisation of Bidirectional Reflectance Distribution Function (BRDF) was carried out by sampling the BRDF space for each material. Consequently, each material is characterised by multiple samples with different incoming and outgoing angles. Finally, an analysis of BRDF sample completeness is performed considering four sunlight positions and 16x16 resolution for each material. The results demonstrated the capability of the used technology and the effectiveness of our method to be used in applications such as spectral rendering and real-word material acquisition and classification.