Search

We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360$^\circ$ RGB video at 15 fps, 3D point clouds from two Velodyne 16 Lidars, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360$^\circ$ spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspective of the robot, both stationary and navigating. The dataset has been annotated with over 2.3 million bounding boxes spread over 5 individual cameras and 1.8 million associated 3D cuboids around all people in the scenes totaling over 3500 time consistent trajectories. Together with our dataset and the annotations, we launch a benchmark and metrics for 2D and 3D person detection and tracking. With this dataset, which we plan on extending with further types of annotation in the future, we hope to provide a new source of data and a test-bench for research in the areas of egocentric robot vision, autonomous navigation, and all perceptual tasks around social robotics in human environments.

In this article, a novel method for simultaneous localization and mapping (SLAM) named surface elements (surfel) point SLAM (SP-SLAM) is proposed using an RGB-D camera. The key idea of SP-SLAM is to use not only keypoints but also surfels as features to cope with both high texture and low texture environments. By decomposing a surface into a small number of surfels, the method can represent spacious environments using a relatively small amount of memory. To optimize the poses of points, surfels, and cameras altogether, new objective functions are proposed, and a new bundle adjustment using these objective functions is developed. The proposed SP-SLAM runs in real time on a central processing unit as in other feature-based visual SLAM methods but works better than them not only in high texture but also in low texture environments, overcoming well-known drawbacks of feature-based visual SLAM with degradation in low texture environments. The proposed method is applied to benchmark datasets and its effectiveness is demonstrated by comparing against those of previous methods in terms of localization accuracy.

Previous neural network accelerators tailored to video analysis only accept data of RGB/YUV domain, requiring decompressing the video that are often compressed before transmitted from the edge sensors. A compressed video processing accelerator can alleviate the decoding overhead, and gain performance speedup by operating on more compact input data. This work proposes a novel deep learning accelerator architecture, Alchemist, which is able to predict results directly from the compressed video bitstream instead of reconstructing the full RGB images. By utilizing the metadata of motion vector and critical blocks extracted from bitstreams, Alchemist contributes to remarkable performance speedup of 5x with negligible accuracy loss. Nevertheless, we still find that the original compressed video coded by standard algorithms such as H.264 is not suitable to be directly manipulated, due to diverse compressed structures. Although obviating the requirement to recover all RGB frames, the accelerator must parse the entire compressed video bitstream to locate reference frames and extract useful metadata. If we combine the video codec with the proposed compressed video analysis, additional optimizations can be obtained. Therefore, to cope with the mismatch between current video coding algorithms such as H.264 and neural network-based video analysis, we propose a specialized coding strategy to generate compressed video bitstreams more suitable for transmission and analysis, which further simplifies the decoding stage of video analysis and is capable of achieving significant storage reduction.

The head-mounted displays (HMDs) provide immersive experiences in virtual reality (VR). However, the face interactions are limited due to the serious occlusion of the user’s face. Existed approaches try to recover the user’s facial expression by adding additional sensors to HMD. In this paper, we develop a novel framework to reconstruct the userb’s 3D face in VR only using an RGB camera. Given a reference face, a realistic full-textured avatar is created by fitting a 3D Morphable Model (3DMM). A self-supervised UV map Generative Adversarial Network (GAN) is proposed to make the facial texture look more realistic. Next, we propose a novel landmark detection method to locate the landmark positions under HMD occlusion since facial landmarks are commonly used for driving the avatar. To this end, we synthesize a face dataset with HMD. Our method is easy to build and popularize with low cost. The experiments on the synthetic and real HMD data demonstrate that the proposed method can detect landmark accurately and restore facial expressions faithfully despite the large occlusion of HMD.

Reconstructing an indoor scene and generating a layout/floor plan in 3D or 2D is a widely known problem. Quite a few algorithms have been proposed in the literature recently. However, most of the existing methods either use RGB-D images, thus requiring a depth camera, or depend on panoramic photos with the assumption that there is little to no occlusion in the rooms. In this work, we proposed generation of layout using an RGB image captured using a simple mobile phone camera. We take advantage of Simultaneous Localization and Mapping (SLAM) to assess the 3D transformations required for layout generation. SLAM technology is built-in in recent mobile libraries such as ARCore by Google. Hence, the proposed method is fast and efficient, while giving the user freedom to generate layout by simply taking a few conventional photos, rather than relying on specialized depth hardware or occlusion-free panoramic photos.

Head pose estimation is an important problem as it facilitates tasks such as gaze estimation and attention modeling. In the automotive context, head pose provides crucial information about the driver's mental state, including drowsiness, distraction and attention. It can also be used for interaction with in-vehicle infotainment systems. While computer vision algorithms using RGB cameras are reliable in controlled environments, head pose estimation is a challenging problem in the car due to sudden illumination changes, occlusions and large head rotations that are common in a vehicle. These issues can be partially alleviated by using depth cameras. Head rotation trajectories are continuous with important temporal dependencies. Our study leverages this observation, proposing a novel temporal deep learning model for head pose estimation from point cloud. The approach extracts discriminative feature representation directly from point cloud data, leveraging the 3D spatial structure of the face. The frame-based representations are then combined with bidirectional long short term memory (BLSTM) layers. We train this model on the newly collected multimodal driver monitoring (MDM) dataset, achieving better results compared to non-temporal algorithms using point cloud data, and state-of-the-art models using RGB images. We further show quantitatively and qualitatively that incorporating temporal information provides large improvements not only in accuracy, but also in the smoothness of the predictions.

Thermal sensors play an important role in intelligent transportation system. This paper studies the problem of RGB and thermal (RGBT) tracking in challenging situations by leveraging multimodal data. A RGBT object tracking method is proposed in correlation filter tracking framework based on short term historical information. Given the initial object bounding box, hierarchical convolutional neural network (CNN) is employed to extract features. The target is tracked for RGB and thermal modalities separately. Then the backward tracking is implemented in the two modalities. The difference between each pair is computed, which is an indicator of the tracking quality in each modality. Considering the temporal continuity of sequence frames, we also incorporate the history data into the weights computation to achieve a robust fusion of different source data. Experiments on three RGBT datasets show the proposed method achieves comparable results to state-of-the-art methods.

Modern vehicles are equipped with various driver-assistance systems, including automatic lane keeping, which prevents unintended lane departures. Traditional lane detection methods incorporate handcrafted or deep learning-based features followed by postprocessing techniques for lane extraction using frame-based RGB cameras. The utilization of frame-based RGB cameras for lane detection tasks is prone to illumination variations, sun glare, and motion blur, which limits the performance of lane detection methods. Incorporating an event camera for lane detection tasks in the perception stack of autonomous driving is one of the most promising solutions for mitigating challenges encountered by frame-based RGB cameras. The main contribution of this work is the design of the lane marking detection model, which employs the dynamic vision sensor. This paper explores the novel application of lane marking detection using an event camera by designing a convolutional encoder followed by the attention-guided decoder. The spatial resolution of the encoded features is retained by a dense atrous spatial pyramid pooling (ASPP) block. The additive attention mechanism in the decoder improves performance for high dimensional input encoded features that promote lane localization and relieve postprocessing computation. The efficacy of the proposed work is evaluated using the DVS dataset for lane extraction (DET). The experimental results show a significant improvement of $5.54\%$ and $5.03\%$ in $F1$ scores in multiclass and binary-class lane marking detection tasks. Additionally, the intersection over union ($IoU$) scores of the proposed method surpass those of the best-performing state-of-the-art method by $6.50\%$ and $9.37\%$ in multiclass and binary-class tasks, respectively.

Skeleton-based human action recognition has received extensive attention due to its efficiency and robustness to complex backgrounds. Though the human skeleton can accurately capture the dynamics of human poses, it fails to recognize human actions induced by the interaction between human and objects, making it is of great importance to further explore the interaction between the human and objects for human action recognition. In this paper, we devise the multi-stream interaction networks (MSIN), to simultaneously explore the dynamics of human skeleton, objects, and the interaction between human and objects. Specifically, apart from the traditional human skeleton stream, 1) the second stream explores the dynamics of object appearance from the objects surrounding the human body joints; and 2) the third stream captures the dynamics of object position in regard to the distance between the object and different human body joints. Experimental results on three popular skeleton-based human action recognition datasets, NTU RGB+D, NTU RGB+D 120, and SYSU, demonstrate the effectiveness of the proposed method, especially for recognizing the human actions with human-object interactions.

RGB salient object detection (SOD) has made great progress. However, the performance of this single-modal salient object detection will be significantly decreased when encountering some challenging scenes, such as low light or darkness. To deal with the above challenges, thermal infrared (T) image is introduced into the salient object detection. This fused modal is called RGB-T salient object detection. To achieve deep mining of the unique characteristics of single modal and the full integration of cross-modality information, a novel Cross-Guided Fusion Network (CGFNet) for RGB-T salient object detection is proposed. Specifically, a Cross-Scale Alternate Guiding Fusion (CSAGF) module is proposed to mine the high-level semantic information and provide global context support. Subsequently, we design a Guidance Fusion Module (GFM) to achieve sufficient cross-modality fusion by using single modal as the main guidance and the other modal as auxiliary. Finally, the Cross-Guided Fusion Module (CGFM) is presented and serves as the main decoding block. And each decoding block is consists of two parts with two modalities information of each being the main guidance, i.e., cross-shared Cross-Level Enhancement (CLE) and Global Auxiliary Enhancement (GAE). The main difference between the two parts is that the GFM using different modalities as the main guide. The comprehensive experimental results prove that our method achieves better performance than the state-of-the-art salient detection methods. The source code has released at: https://github.com/wangjie 0825/CGFNet.git.

RGB-T salient object detection (SOD) aims at utilizing the complementary cues of RGB and Thermal (T) modalities to detect and segment the common objects. However, on one hand, existing methods simply fuse the features of two modalities without fully considering the characters of RGB and T. On the other hand, the high computational cost of existing methods prevents them from real-world applications (e.g., automatic driving, abnormal detection, person re-ID). To this end, we proposed an efficient encoder-decoder network named Context-guided Stacked Refinement Network (CSRNet). Specifically, we utilize a lightweight backbone and design efficient decoder parts, which greatly reduce the computational cost. To fuse RGB and T modalities, we proposed an efficient Context-guided Cross Modality Fusion (CCMF) module to filter the noise and explore the complementation of two modalities. Besides, Stacked Refinement Network (SRN) progressively refines the features from top to down via the interaction of semantic and spatial information. Extensive experiments show that our method performs favorably against state-of-the-art algorithms on RGB-T SOD task while with small model size (4.6M), few FLOPs (4.2G), and real-time speed (38fps). Our codes is available at: https://github.com/huofushuo/CSRNet.

International audience; Several works have focused on Simultaneous Localization and Mapping (SLAM), which is a topic that has been studied for more than a decade to meet the needs of robots to navigate in an unknown environment. SLAM is an essential perception functionality in several applications, especially in robotics and autonomous vehicles. RGB-D cameras are among the sensors commonly used with recent SLAM algorithms. They provide an RGB image and the associated depth map, making it possible to solve scale drift with less complexity and create a dense 3D environment representation. Many RGB-D SLAM algorithms have been studied and evaluated on publicly available datasets without considering sensor specifications or image acquisition modes that could improve or decrease localization accuracy. In this work, we deal with indoor localization, taking into account the sensor specifications. In this context, our contribution is a deep experimental study to highlight the impact of the sensor acquisition modes on the localization accuracy, and a parametric optimization protocol for a precise localization in a given environment. Furthermore, we apply the proposed protocol to optimize a depth-related parameter of the SLAM algorithm. The study is based on a publicly available dataset in an indoor environment with a depth sensor. The reconstruction results’ analysis is founded on the study of different metrics involving translational and rotational errors. These metrics errors are compared with those obtained with a state-of-the-art stereo vision-based SLAM algorithm.

Recent years have witnessed the prevalence of using the Siamese network for RGB-T tracking because of its remarkable success in RGB object tracking. Despite their faster than real-time speeds, existing RGB-T Siamese trackers suffer from low accuracy and poor robustness, compared to other state-of-the-art RGB-T trackers. To address such issues, a new complementarity- and distractor-aware RGB-T tracker based on Siamese network (referred to as SiamCDA) is developed in this paper. To this end, several modules are presented, where the feature pyramid network (FPN) is incorporated into the Siamese network to capture the cross-level information within unimodal features extracted from the RGB or the thermal images. Next, a complementarity-aware multi-modal feature fusion module (CA-MF) is specially designed to capture the cross-modal information between RGB features and thermal features. In the final bounding box selection phase, a distractor-aware region proposal selection module (DAS) further enhances the robustness of our tracker. On top of the technical modules, we also build a large-scale, diverse synthetic RGB-T tracking dataset, containing more than 4831 pairs of synthetic RGB-T videos and 12K synthetic RGB-T images. Extensive experiments on three RGB-T tracking benchmark datasets demonstrate the outstanding performance of our proposed tracker with a tracking speed over 37 frames per second (FPS).

International audience; The goal of Mixed Reality (MR) is to achieve a seamless and realistic blending between real and virtual worlds. This requires the estimation of reflectance properties and lighting characteristics of the real scene. One of the main challenges within this task consists in recovering such properties using a single RGB-D camera. In this paper, we introduce a novel framework to recover both the position and color of multiple light sources as well as the specular reflectance of real scene surfaces. This is achieved by detecting and incorporating information from both specular reflections and cast shadows. Our approach is capable of handling any textured surface and considers both static and dynamic light sources. Its effectiveness is demonstrated through a range of applications including visually-consistent mixed reality scenarios (e.g. correct real specularity removal, coherent shadows in terms of shape and intensity) and retexturing where the texture of the scene is altered whereas the incident lighting is preserved.

Prevalent video codec implementations deployed in the field often solely support chrominance sub-sampled video data represented by the YCbCr 4:2:0 format. For certain applications like screen sharing, however, chroma sub-sampling leads to disturbing artifacts, especially for text or graphics with thin lines. It is desirable to reduce these artifacts while maintaining compatibility with all user devices. For this reason, an enhancement layer coding framework for chroma-format scalable coding with specific focus on screen content is proposed in this paper. Based on an analysis of screen content data characteristics, the enhancement layer codec is optimized specifically for this content class, is of low algorithmic complexity, and applicable with any image or video codec for base layer compression in a joint coding system. The system is intentionally not designed as a general purpose lossless YCbCr 4:4:4 coding scheme, but instead chooses to close a quality gap that prevalent video codecs do not address. Experimental analysis reveals an average BD-PSNR gain in the RGB domain of 1.0 dB comparing the proposed two-layer scalable coding approach to single layer compression using base layer coding only. In relation to simulcast coding, an average BDR RGB difference of 8.1% is observed.

Camera and LIDAR are both important sensor modalities for real-world applications, especially autonomous driving. The sensors provide complementary information and make sensor fusion possible. However, the progress of early-fusion for both modalities is very slow due to the limitations of viewpoint misalignment, feature misalignment and data volume alignment, so that its performance is also very low. In this work, we propose a novel fusion pipeline: an early-fusion method of range image and RGB image to enhance 3D object detection. It takes full advantage of LIDAR’s range view, point view, bird’s eye view, and RGB view of the camera: First, it uses a large-scale 2D detection data set to pre-train a 2D detection network for extracting high-level semantic features from the RGB image; Second, it uses a specially designed 2D convolution network to extract high-level geometric features from the range image; Third, it fuses the semantic and geometric features through the point view of point clouds; Finally, the point view is transferred to bird’s eye view for 3D object detection. Both the range image and the RGB image are front views, so the multi-modal features are more matched. Experiments show that the proposed method has large improvements on the method of using only the point cloud, and outperforms the state-of-the-art methods on the self-built actual data set. In ablation, we study the dependence of the fusion method on 2D features and the effect of different fusion positions on the performance.

The coregistration of terrestrial laser point clouds is widely investigated where different techniques are presented to solve this problem. The techniques are divided either as target-based or targetless approaches for coarse and fine coregistration. The targetless approach is more challenging since no physical reference targets are placed in the field during the scanning. Mainly, targetless methods are image-based and they are applied through projecting the point clouds back to the scanning stations. The projected 360 point cloud images are normally in the form of panoramic images utilizing either intensity or RGB values, and an image matching is followed to align the scan stations together. However, the point cloud coregistration is still a challenge since ICP like methods are applicable for fine registration. Furthermore, image-based approaches are restricted when there is: a limited overlap between point clouds, no RGB data accompanied to intensity values, and unstructured scanned objects in the point clouds. Therefore, we present in this paper the concept of a multi surrounding scan MSS image-based approach to overcome the difficulty to register point clouds in challenging cases. The multi surrounding scan approach means to create multi-perspective images per laser scan point cloud. These multi-perspective images will offer different viewpoints per scan station to overcome the viewpoint distortion that causes the failure of the image matching in challenging situations. Two experimental tests are applied using point clouds collected in Enschede city and the published 3D toolkit data set in Bremen city. The experiments showed a successful coregistration approach even in challenging settings with different constellations.

This study aims at developing and evaluating an affordable and user-friendly motion capture system for human upper limbs' joint kinematics estimation. The objective is to provide quantitative assessments during the clinical evaluation of poststroke patients performing daily living activities. The proposed system is based on the simultaneous use of affordable inertial measurement units, and a set of augmented reality markers tracked with an affordable RGB camera. Two practical calibration processes were developed to calibrate the sensors modules and then to determine their location on body segments. Then, all measured quantities were fused into a constrained extended Kalman filter based on an upper limbs' biomechanical model. The proposed system was validated with nine healthy volunteers performing five daily living activities. Joint angles estimated using the proposed affordable system were compared with a gold standard stereophotogrammetric system. The results showed a low average rms difference (2.7°) along with a high average correlation (0.87).

Integrating multispectral data has been demonstrated to be an effective solution for illumination-invariant pedestrian detection, in particular, RGB and thermal images can provide complementary information to handle light variations. However, most of the current multispectral detectors fuse the multimodal features by simple concatenation, without discovering their latent relationships. In this paper, we propose a cross-modal feature learning (CFL) module, based on a split-and-aggregation strategy, to explicitly explore both the shared and modalityspecific representations between paired RGB and thermal images. We insert the proposed CFL module into multiple layers of a twobranch-based pedestrian detection network, to learn the crossmodal representations in diverse semantic levels. By introducing a segmentation-based auxiliary task, the multimodal network is trained end-to-end by jointly optimizing a multi-task loss. On the other hand, to alleviate the reliance of existing multispectral pedestrian detectors on thermal images, we propose a knowledge distillation framework to train a student detector, which only receives RGB images as input and distills the cross-modal representations guided by a well-trained multimodal teacher detector. In order to facilitate the cross-modal knowledge distillation, we design different distillation loss functions for the feature, detection and segmentation levels. Experimental results on the public KAIST multispectral pedestrian benchmark validate that the proposed cross-modal representation learning and distillation method achieves robust performance.

In this paper we present an approach to jointly recover camera pose, 3D shape, and object and deformation type grouping, from incomplete 2D annotations in a multi-instance collection of RGB images. Our approach is able to handle indistinctly both rigid and non-rigid categories. This advances existing work, which only addresses the problem for one single object or, they assume the groups to be known a priori when multiple instances are handled. In order to address this broader version of the problem, we encode object deformation by means of multiple unions of subspaces, that is able to span from small rigid motion to complex deformations. The model parameters are learned via Augmented Lagrange Multipliers, in a completely unsupervised manner that does not require any training data at all. Extensive experimental evaluation is provided in a wide variety of synthetic and real scenarios, including rigid and non-rigid categories with small and large deformations. We obtain state-of-the-art solutions in terms of 3D reconstruction accuracy, while also providing grouping results that allow splitting the input images into object instances and their associated type of deformation., This work has been partially supported by the CSIC project R3OBJ 201850I099, the Spanish Ministry of Science and Innovation under project HuMoUR TIN2017-90086-R as well as the Salvador de Madariaga grant PRX19/00626.

Compared with traditional handcrafted features, deep learning has greatly improved the performance of scene parsing. However, it remains challenging under various environmental conditions caused by imaging limitations. Thermal imaging cameras have several advantages over cameras for the visible spectrum, such as operation in total darkness, robustness to shadow effects, insensitivity to illumination variations, and strong ability to penetrate smog and haze. These advantages of thermal imaging cameras make them ideal for the scene parsing of semantic objects in daytime and nighttime. In this paper, we propose a novel multiscale feature fusion and enhancement network (MFFENet) for accurate parsing of RGBthermal urban road scenes even when the quality of the available RGB data is compromised. The proposed MFFENet consists of two encoders, a feature fusion layer, and a multi-labelsupervision layer. We concatenate the multi-scale features with the features that contain global semantic information. Furthermore, we explore the cross-modal fusion of RGB and thermal features at multiple stages, rather than fusing them once at the low or high stage. Then, we propose a spatial attention mechanism module that provides a higher weight to (focuses more on) the foreground area, allowing MFFENet to emphasize foreground objects. Finally, multi-labelsupervision is introduced to optimize parameters of the proposed MFFENet. Experimental results confirm that the proposed MFFENet outperforms similar high-performing methods.

Visual odometry (VO) is a fundamental building block for many robotics and augmented reality (AR) applications. However, most existing RGB-D VO systems could suffer from large performance degradation when large occlusions are present or a large portion of depth values are invalid due to the limited range of an RGB-D camera, prohibiting the usage of most systems in practical applications. In this paper, we present RGB-D DSO, a RGB-D direct sparse odometry which can achieve state-of-theart accuracy, robustness and efficiency in the indoor scenes. Our RGB-D DSO is a significant extension of monocular DSO [1], with the core part being sliding-window optimization with occlusion removal and a depth refinement module. Occlusion removal excludes negative effects arising from occluded objects when minimizing the final energy function for camera pose tracking. Depth refinement ensures sufficient valid depth values uniformly distributed for the depth map of a keyframe. Experimental results on three public datasets [2], [3], [4] demonstrate that our method achieves smaller absolute trajectory errors (ATE) and relative pose error (RPE) than the state-of-the-art methods DVO [5], ORB-SLAM2 [6], CVO[7] and ACVO[8]. Meanwhile, our system takes only 21.93ms to track a frame, which is 3.7X faster than DVO (80.27ms/frame) and 1.4X faster than ORB-SLAM2 (29.91ms/frame).

Unmanned aerial vehicles (UAVs) allow on-demand imaging of orchards at an unprecedented level of detail. The automated detection of plantation rows in the images helps in the successive analysis steps, such as the detection of individual fruit trees and planting gaps, aiding producers with inventory and planting operations. Citrus trees can be planted in curved rows that form intricate geometric patterns in aerial images, requiring robust detection approaches. While deep learning methods rank among state-of-the-art methods for segmenting images with particular geometrical patterns, they struggle to hold their performance when testing data differs much from training data (e.g., image intensity differences, image artifacts, vegetation characteristics, and landscape conditions). In this letter, we propose a method to learn geometric features of orchards in UAV images and use them to improve the detection of plantation rows. First, we train a detection encoder-decoder network (DetED) to segment planting rows in RGB images. Then, with labeled data, we train an encoder-decoder correction network (CorrED) that learns to map binary masks with spurious row segmentation geometries into corrected ones. Finally, we use the CorrED network to fix geometric inconsistencies in DetED outcome. Our experiments with commercial plantations of orange trees show that the proposed CorrED postprocessing can restore missing segments of plantation rows and improve detection accuracy in testing data.

This letter presents a robust, real-time visual odometry that efficiently exploits the available visual and geometry cues from RGB-D frames for both tracking and mapping. Together with a hybrid tracking algorithm based on a joint multi-objective formulation, we additionally incorporate the point-to-plane metrics into the photometric bundle adjustment (PBA) to constrain the iteration direction especially for those weakly-textured points. The relative pose constraints derived from optimized poses via PBA is then leveraged in combination with reprojection constraints retrieved from maintained feature map, to refine keyframe poses and feature locations. Moreover, the slanted support plane commonly used in multi-view stereo matching, is utilized for the adjustment of the semi-dense points to further enhance mapping accuracy that in turn benefits the front-end tracking. We extensively evaluate our algorithm on benchmark datasets, and those experimental results validate the advantage of our method in terms of overall tracking performance over other representative approaches.

6D object pose estimation for texture-less objects from RGB images remains challenging, especially in occlusion scenarios. Instead of localizing sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion, we introduce GeoPose, a novel reconstruction guided pose estimation pipeline that predicts dense correspondences and leverages geometric consistency effectively. We first design a dense reconstruction network (ReconNet) to reconstruct pixel-wise object coordinates in normalization space. Dense 2D-3D correspondences are generated intuitively by our explicit pa-rameterization for 3D object models, which dismisses keypoint selection efforts. These 2D-3D correspondences are then utilized to estimate 6D poses by the PnP algorithm with RANSAC iterations. Furthermore, a novel Cycle Loss is proposed to provide 3D prior supervision, which significantly correlates with the pose estimation task by guiding geometric consistency between reconstruction (pixel to 3D) and reprojection (3D to pixel). In addition, a training data augmentation method is proposed to handle the insufficiency of 6D datasets, the acquisition of which is error-prone and time-consuming. Extensive experiments demonstrate that, compared with existing RGB-based methods, our GeoPose can achieve state-of-the-art (SOTA) 6D pose estimation performance on the LINEMOD, Occlusion LINEMOD and T-LESS datasets.

Salient Object Detection (SOD) plays an important role in many image-related multimedia applications. Although there are many existing research works about the salient object detection in traditional RGB (visible-light spectrum) images, there are still many complex situations that regular RGB images cannot provide enough cues for the accurate SOD, such as the shadow effect, similar appearance between background and foreground, strong or insufficient illumination, etc. Because of the success of near-infrared spectrum in many computer vision tasks, we explore the multi-spectral SOD in the synchronized RGB images and near-infrared (NIR) images for the both simple and complex situations. We assume that the RGB SOD in the existing RGB image datasets could provide references for the multi-spectral SOD problem. In this paper, we mainly model this research problem as a deep learning based domain adaptation from the traditional RGB image data (source domain) to the multi-spectral data (target domain), and an adversarial deep domain adaptation model is proposed. We first collect and will publicize a large multi-spectral dataset, RGBN-SOD dataset, including 780 synchronized RGB and NIR image pairs for the multi-spectral SOD problem in the simple and complex situations. Intensive experimental results show the effectiveness and accuracy of the proposed deep domain adaptation for the multi-spectral SOD. Besides, due to the absence of research on the field of multi-spectral co-saliency detection, we also collect 200 synchronized RGB and NIR image pairs in addition to explore the multi-spectral co-saliency detection.

In this study, we propose a multimodal feature based framework for recognising hand gestures from RGB and depth images. In addition to the features from the RGB image, the depth image features are explored into constructing the discriminative feature labels of various gestures. Depth maps having powerful source of information, increases the performance level of various computer vision problems. A newly refined Gradient-Local Binary Pattern (G-LBP) is applied to extract the features from depth images and histogram of gradients (HOG) features are extracted from RGB images. The components from both RGB and depth channels, are concatenated to form a multimodal feature vector. In the final process, classification is performed using K-Nearest Neighbour and multi-class Support Vector Machines. The designed system is invariant to scale, rotation and illumination. The newly developed feature combination method is helpful to achieve superior recognition rates for future innovations.

Thermal infrared (TIR) images acquired from Unmanned Aircraft Vehicles (UAV) are gaining scientific interest in a wide variety of fields. However, the reconstruction of three-dimensional (3D) point clouds utilizing consumer-grade TIR images presents multiple drawbacks as a consequence of low-resolution and induced aberrations. Consequently, these problems may lead photogrammetric techniques, such as Structure from Motion (SfM), to generate poor results. This work proposes the use of RGB point clouds estimated from SfM as the input for building thermal point clouds. For that purpose, RGB and thermal imagery are registered using the Enhanced Correlation Coefficient (ECC) algorithm after removing acquisition errors, thus allowing us to project TIR images into an RGB point cloud. Furthermore, we consider several methods to provide accurate thermal values for each 3D point. First, the occlusion problem is solved through two different approaches, so that points that are not visible from a viewing angle do not erroneously receive values from foreground objects. Then, we propose a flexible method to aggregate multiple thermal values considering the dispersion from such aggregation to the image samples. Therefore, it minimizes error measurements. A naive classification algorithm is then applied to the thermal point clouds as a case study for evaluating the temperature of vegetation and ground points. As a result, our approach builds thermal point clouds with up to 798,69% more point density than results from other commercial solutions. Moreover, it minimizes the build time by using parallel computing for time-consuming tasks. Despite obtaining larger point clouds, we report up to 96,73% less processing time per 3D point.

Recognizing various human actions in video is a challenging task, which is also one of the key tasks in computer vision. It has received extensive attention from AI researchers Bakker et al. (2003), Bruderlin and Williams (1995), Cardle et al. (2003), Carlsson (1996, 1999), Clausen and Kurth (2004). It has important applications in human behavior analysis, artificial intelligence, and video surveillance. Compared with still image classification, the time component of video provides important clues for recognition, so multiple human actions can be recognized based on motion information. In addition, video provides natural data enhancement for individual images. For motion recognition from videos, appearance and temporal dynamics are two key and complementary cues. In this work, we formulate a human motion recognition framework based on 2D spatial feature fusion of Kinect bone data. This method utilizes the human body structure and spatial geometry to represent human body structure and extract features. Meanwhile, it combines the active action and auxiliary action features of two spatial dimensions by layering, and leverages the pervasively used support vector machine and hidden Markov model to classify human body structure. Noticeably, it is difficult to extract its information due to the limitations of background clutter, viewpoint change, scale change, different lighting conditions and camera movement. In this way, while learning the classification information of human behavioral categories, designing effective representations is the key to deal with these challenges. In this work, a method for human motion recognition in video based on ResNext network is proposed. Based on our proposed ResNeXt network, using the data of RGB and optical flow, we can extract more appearance characteristics and temporal features of human actions, so as to better achieve the classification of actions. The utilization of video time segmentation method can capture the long range of time in video, in order to make better use of the longer range of time information in video. Comprehensive experimental results have shown that the proposed method improves the performance of UCF101 and HMDB51 motion recognition data sets to a certain extent.

Medical systems and assistive technologies, human-computer interaction, human-robot interaction, industrial automation, virtual environment control, sign language translation, crisis and disaster management, entertainment and computer games, and so on all use RGB cameras for hand gesture recognition. However, their performance is limited especially in low-light conditions. In this paper, we propose a robust hand gesture recognition system based on high-resolution thermal imaging that is light-independent. A dataset of 14,400 thermal hand gestures is constructed, separated into two color tones. We also propose using a deep CNN to classify high-resolution hand gestures accurately. The proposed models were also tested on Raspberry Pi 4 and Nvidia AGX edge computing devices, and the results were compared to the benchmark models. The model also achieves an accuracy of 98.81% and an inference time of 75.138 ms on Nvidia Jetson AGX. In contrast to hand gesture recognition systems based on RGB cameras, which have limited performance in the dark-light conditions, the proposed system based on reliable high resolution thermal images is well-suited to a wide range of applications.

To solve the problem that standard image decomposition cannot conduct image decomposition of basketball shooting angle according to the change of basketball projection Angle, an intelligent visual image decomposition method of basketball projection angle using odd-difference value decomposition was proposed. By analyzing the change of basketball shooting angle under different shooting conditions, the basketball shooting angle is determined. Based on this, singular value decomposition is introduced to judge the image decomposition of intelligent vision. Observations have shown that the intelligent vision image decomposition judgment method can improve the image contrast of shooting motion decomposition. In the shortest time, according to the change of basketball projection angle, the shooting motion image decomposition is fulfilled. In order to improve the accuracy of human motion recognition in videos and computation of large-scale data sets, a motion recognition technique by leveraging double-core extreme learning machine and deep learning technology are proposed. In the first layer of the double-core extreme learning component, the linear kernel extreme learning machine is used to predict the dense trajectory characteristics and deep learning characteristics. In the second layer, dense trajectory features are combined with deep learning features. In deep features, deep video features are fused with video RGB trichromatic features as deep features. Simulation experiments are conducted on a massive-scale and real-world data sets and a small basketball gesture data set. The experimental results have shown that our method exhibits a high recognition accuracy for both data sets. The analysis results have shown that the improved model has higher recognition rate and better action recognition effect than its competitors.

In this work, we propose a strong two-stream baseline method referred to as GeometryMotion-Net for 3D action recognition. For efficient 3D action recognition, we first represent each point cloud video as a limited number of randomly sampled frames with each frame consisting of a sparse set of points. After that, we propose a new two-stream framework for effective 3D action recognition. For the geometry stream, we propose a new module to produce a virtual overall geometry point cloud by first merging all 3D points from these selected frames, and then we exploit local neighborhood information of each point in the feature space. In the motion stream, for any two neighboring point cloud frames, we also propose a new module to generate one virtual forward motion point cloud and one virtual backward motion point cloud. Specifically, for each point in the current frame, we first produce a set of 3D offset features relative to the neighboring points in the reference frame (i.e., the previous/subsequent frame) and then exploit local neighborhood information of this point in the offset feature space. Based on the newly generated virtual overall geometry point cloud and multiple virtual forward/backward motion point clouds, any existing point cloud analysis methods (e.g., PointNet) can be readily adopted for extracting discriminant geometry and bidirectional motion features in the geometry and motion streams, respectively, which are further aggregated to make our two-stream network trainable in an end-to-end fashion. Comprehensive experiments on both large-scale datasets (i.e. NTU RGB+D 60 and NTU RGB+D 120) and small-scale datasets (i.e., N-UCLA and UWA3DII) demonstrate the effectiveness and efficiency of our two-stream network for 3D action recognition.

6D object pose estimation plays a crucial role in robotic manipulation and grasping tasks. The aim to estimate the 6D object pose from RGB or RGB-D images is to detect objects and estimate their orientations and translations relative to the given canonical models. RGB-D cameras provide two sensory modalities: RGB and depth images, which could benefit the estimation accuracy. But the exploitation of two different modality sources remains a challenging issue. In this paper, inspired by recent works on attention networks that could focus on important regions and ignore unnecessary information, we propose a novel network: Channel-Spatial Attention Network (CSA6D) to estimate the 6D object pose from RGB-D camera. The proposed CSA6D includes a pre-trained 2D network to segment the interested objects from RGB image. Then it uses two separate networks to extract appearance and geometrical features from RGB and depth images for each segmented object. Two feature vectors for each pixel are stacked together as a fusion vector which is refined by an attention module to generate a aggregated feature vector. The attention module includes a channel attention block and a spatial attention block which can effectively leverage the concatenated embeddings into accurate 6D pose prediction on known objects. We evaluate proposed network on two benchmark datasets YCB-Video dataset and LineMod dataset and the results show it can outperform previous state-of-the-art methods under ADD and ADD-S metrics. Also, the attention map demonstrates our proposed network searches for the unique geometry information as the most likely features for pose estimation. From experiments, we conclude that the proposed network can accurately estimate the object pose by effectively leveraging multi-modality features.

We present a sample of 16 likely strong gravitational lenses identified in the VST Optical Imaging of the CDFS and ES1 fields (VOICE survey) using Convolutional Neural Networks (CNNs). We train two different CNNs on composite images produced by superimposing simulated gravitational arcs on real Luminous Red Galaxies observed in VOICE. Specifically, the first CNN is trained on single-band images and more easily identifies systems with large Einstein radii, while the second one, trained on composite RGB images, is more accurate in retrieving systems with smaller Einstein radii. We apply both networks to real data from the VOICE survey, taking advantage of the high limiting magnitude (26.1 in the r-band) and low PSF FWHM (0.8" in the r-band) of this deep survey. We analyse $\sim21,200$ images with $mag_r, 15 pages, 12 figures, submitted to MNRAS

Hand gesture recognition is a challenging topic in the field of computer vision. Multimodal hand gesture recognition based on RGB-D is with higher accuracy than that of only RGB or depth. It is not difficult to conclude that the gain originates from the complementary information existing in the two modalities. However, in reality, multimodal data are not always easy to acquire simultaneously, while unimodal RGB or depth hand gesture data are more general. Therefore, one hand gesture system is expected, in which only unimordal RGB or Depth data is supported for testing, while multimodal RGB-D data is available for training so as to attain the complementary information. Fortunately, a kind of method via multimodal training and unimodal testing has been proposed. However, unimodal feature representation and cross-modality transfer still need to be further improved. To this end, this paper proposes a new 3D-Ghost and Spatial Attention Inflated 3D ConvNet (3DGSAI) to extract high-quality features for each modality. The baseline of 3DGSAI network is Inflated 3D ConvNet (I3D), and two main improvements are proposed. One is 3D-Ghost module, and the other is the spatial attention mechanism. The 3D-Ghost module can extract richer features for hand gesture representation, and the spatial attention mechanism makes the network pay more attention to hand region. This paper also proposes an adaptive parameter for positive knowledge transfer, which ensures that the transfer always occurs from the strong modality network to the weak one. Extensive experiments on SKIG, VIVA, and NVGesture datasets demonstrate that our method is competitive with the state of the art. Especially, the performance of our method reaches 97.87% on the SKIG dataset using only RGB, which is the current best result.

The equidistance of the color space plays a significant role in determining the color difference in color transmission systems. Strictly equal contrasting color spaces can be considered only those color spaces in which equal changes in the visual perception of color are provided with an equal change in the color coordinates in this color space. Currently, the International Commission on Lighting (CIE) has adopted a number of color spaces called equal-contrast. The article presents the results of the study of color spaces adopted by CIE for equal contrast, i.e. on the differences in the thresholds of color differentiation in different areas of the color locus. The article investigated such color spaces as CIE 1931 (RGB), CIE 1931 (x, y), CIE 1960 (u, v), CIE 1976 (u*, v*), CIE LAB (a*, b*).

As augmented reality technologies develop, real-time interactions between objects present in the real world and virtual space are required. Generally, recognition and location estimation in augmented reality are carried out using tracking techniques, typically markers. However, using markers creates spatial constraints in simultaneous tracking of space and objects. Therefore, we propose a system that enables camera tracking in the real world and visualizes virtual visual information through the recognition and positioning of objects. We scanned the space using an RGB-D camera. A three-dimensional (3D) dense point cloud map is created using point clouds generated through video images. Among the generated point cloud information, objects are detected and retrieved based on the pre-learned data. Finally, using the predicted pose of the detected objects, other information may be augmented. Our system estimates object recognition and 3D pose based on simple camera information, enabling the viewing of virtual visual information based on object location.

This paper investigates the multi-sensor fusion perceptual target recognition based on machine learning visual feature extraction. On the basis of the efficient neural network model (Enet), an improved efficient semantic segmentation network model (Enet-CRF) is designed according to the requirements of the research content. This network model merges a CRF-RNN back-end optimization network to the original Enet, and further improves the classification performance of the network by adding constraints on the position relationship between image pixels and RGB information. The experiments indicate that the designed Enet-CRF has a certain improvement in obstacle classification performance, especially for small-scale obstacles such as pedestrians and bicycles. By deep fusion of LiDAR and vision sensors at the data level and higher-order features, this method adds high-precision radar information to the original Enet-CRF network model, thereby further enhancing the accuracy of obstacle classification and achieving an obstacle recognition method. This approach can not only accurately classify obstacles, but also attain accurate spatial information of obstacles, with good real-time performance.

Distinguishing target object under occlusions has become the forefront of research to cope with grasping study in general. In this paper, a novel framework which is able to be utilized for a parallel robotic gripper is proposed. There are two key steps for the proposed method in the process of grasping occluded object: generating template information and grasp detection using the matching algorithm. A neural network, trained by the RGB-D data from the Cornell Grasp Dataset, predicts multiple grasp rectangles on template images. A proposed matching algorithm is utilized to eliminate the influence caused by occluded parts on scene images and generates multiple grasp rectangles for objects under occlusions using the grasp information of matched template images. In order to improve the quality of matching result, the proposed matching algorithm improves the SIFT algorithm and combines it with the improved RANSAC algorithm. In this way, this paper obtains suitable grasp rectangles on scene images and offers a new thought about grasping detection under occlusions. The validation results show the effectiveness and efficiency of this approach.

Recently, pose estimation algorithms have been widely discussed. Although been investigated by many prior works, pose estimation for heavily stacked workpieces, like bins, is still a challenge for industrial applications. Moreover, the scene of stacked workpieces is much more arduous than the general scene for robots to carry out routine tasks, such as object detection and picking. This paper aims to address the problem of pose estimation for stacked symmetrical workpieces in the presence of partial occlusions and cluttered backgrounds. To tackle those problems, we propose a novel pose estimation method for industrial stacking scenes based on RGB images. Specifically, due to the common symmetry characteristics of industrial workpieces, we firstly present a new standardized spatial representation method, which could auto-encode the 2D-3D correspondences of symmetrical workpieces. Besides, we introduce a novel GAN-based deep neural network model to reconstruct the representation of stacked workpieces. Based on that, the pose of the target workpieces is predicted based on the reconstructed expression and an improved RANSAC-PnP algorithm. Finally, comprehensive experiments demonstrate that the proposed method outperforms state-of-the-art methods, especially in complex stacking scenes.

Pixel value access control (PVAC) was introduced to deliver a secure and simple graphical password method where it requires users to load their image as their password. PVAC extracts the image to obtain a three-octet 8-bits Red-Green-Blue (RGB) value as its password to authenticate a user. The pixel value must be matched with the record stored in the database or otherwise, the user is failed to authenticate. However, users which prefer to store images on cloud storage would unintentionally alter and as well as the pixel value due to media compression and caused faulty pixels. Thus, the K-Means clustering algorithm is adapted to fix the issue where the faulty pixel value would be recognized as having the same pixel value cluster as the original. However, most of K-Means algorithm works were mainly developed for content-based image retrieval (CBIR) which having opposite characteristics from PVAC. Thus, this study was aimed to investigate the crucial criteria of PVAC and its compatibility with the K-Means algorithm for the problem. The theoretical analysis is used for this study where the suitable characteristics of K-Means are analyze based on PVAC requirements. The compliance analysis might become a referencing work for digital image clustering techniques adaptation on security system such as image filtering, image recognition, and object detection since most of image clustering works was focused on less sensitive image retrieval.