Search

Visual perception is a crucial component of autonomous driving systems. Traditional approaches for autonomous driving visual perception often rely on single-modal methods, and semantic segmentation tasks are accomplished by inputting RGB images. However, for semantic segmentation tasks in autonomous driving visual perception, a more effective strategy involves leveraging multiple modalities, which is because different sensors of the autonomous driving system bring diverse information, and the complementary features among different modalities enhance the robustness of the semantic segmentation modal. Contrary to the intuitive belief that more modalities lead to better accuracy, our research reveals that adding modalities to traditional semantic segmentation models can sometimes decrease precision. Inspired by the residual thinking concept, we propose a multimodal visual perception model which is capable of maintaining or even improving accuracy with the addition of any modality. Our approach is straightforward, using RGB as the main branch and employing the same feature extraction backbone for other modal branches. The modals score module (MSM) evaluates channel and spatial scores of all modality features, measuring their importance for overall semantic segmentation. Subsequently, the modal branches provide additional features to the RGB main branch through the features complementary module (FCM). Leveraging the residual thinking concept further enhances the feature extraction capabilities of all the branches. Through extensive experiments, we derived several conclusions. The integration of certain modalities into traditional semantic segmentation models tends to result in a decline in segmentation accuracy. In contrast, our proposed simple and scalable multimodal model demonstrates the ability to maintain segmentation precision when accommodating any additional modality. Moreover, our approach surpasses some state-of-the-art multimodal semantic segmentation models. Additionally, we conducted ablation experiments on the proposed model, confirming that the application of the proposed MSM, FCM, and the incorporation of residual thinking contribute significantly to the enhancement of the model.

Abstract Stereo image pairs can effectively enhance the performance of super‐resolution (SR) since both intra‐view and cross‐view information can be used. However, exploiting cross‐view information accurately is extremely challenging. Most recent methods use the attention mechanism to get stereo correspondence. But these methods ignore the high‐frequency information since they only utilise first‐order statistics, which leads to reducing the discriminative ability of the network. To address this issue, in this work, a parallax‐based second‐order mixed attention stereo SR network (PSMASSRnet) is proposed to integrate the cross‐view information from a stereo image pair for SR. Specifically, a novel parallax‐based second‐order mixed attention module (PSMAM) is developed to combine second‐order channel features and spatial features to obtain more discriminative representations. Furthermore, a dense cross‐atrous spatial pyramid pooling (ASPP) module is also presented, which can effectively explore the local and the multi‐scale features with different dilation rates to extract more discriminative features with fewer parameters and less execution. The extensive experiments on the KITTI2012, KITTI2015 and Middlebury datasets have demonstrated the superiority of the proposed PSMASSRnet over the state‐of‐the‐art methods in the aspects of both the quantitative metrics and the visual quality.

In the research fields of 3D animation, virtual reality, etc., it is a very important task to reconstruct a 3D human body model with the surface textures. However, due to factors such as diversities of human body poses, uncertainty of depth, and a wide range of viewpoints, the reconstruction of the 3D human body from a single image is an enormous challenge. To solve the aforementioned challenge, this paper proposes a novel architecture to reconstruct the 3D human mesh with great surface details. The key point of the proposed architecture is to combine the topology of the SMPL (Skinned Multi-Person Linear) template mesh and the flexible mesh deformation. Firstly, the typical 2D CNN architecture, ResNet-50, extracts the perceptual features from the input image. The image features are fused into the 3D space via perceptual feature transformation. Our approach does not rely on the parameter space, although the mesh topology of the SMPL is preserved. The approach fully exploits the ability of the SMPL to represent various human body shapes, while avoiding the problem of the spatial loss caused by the parametric method. Secondly, to recover more details of the human body, a graph-based convolutional network is adopted to deform the 3D human body mesh. The network allows starting with fewer vertices and distributing vertex features to the representative positions in a coarse-to-fine manner, which fully utilizes the mesh topology. Experiments demonstrate that our approach can generate 3D human body mesh with great details qualitatively. Compared with the state-of-the-art methods, our approach also achieves better accuracy quantitatively.

The main purpose of chat robots is to realize intelligent interaction between human beings and chat robots. Generally, a complete conversation involves chat contexts. Before responding, human beings need to extract information from their chat contexts, and human beings are very good at extracting information. However, how to make the chatbots more complete in extracting appropriate contextual information has become a key issue for the chatbots in multi-round dialogues. Most research in recent years has paid attention to the point-to-point matching of responses and utterances at reciprocal granularity, such as SMN (Sequential Matching Network) and DAM (Deep Attention Matching Network), for which these parsing methods affect the effectiveness of the above networks to some extent. For example, DAM introduces an attention mechanism, which can obtain good results by parsing five levels of granularity of response and utterance through a self-attention module, and then matching the same level of granularity to mine similar spatial information in it. Based on the structure of DAM, this paper proposes an information-mining idea for improving DAM, which applies to those models with a multi-layer matching structure. Therefore, based on this idea, this paper presents two improved methods for DAM, so as to improve the accuracy of the information extraction from the contexts of the multiple round chats. The experiments show that the improved DAM has better results than that of the unimproved DAM, which are, respectively, R_2@1 increased by 0.425%, R_10@1 increased by 0.515%, R_10@2 increased by 0.341%, MAP (Mean Average Precision) increased by 0.358% in Douban data set, P@1 increased by 1.16%, R_10@1 increased by 1.17%, and these results are better than those state-of-the-art methods, and the improved methods presented in this paper can be used not only in DAM but also in other models with similar the point-to-point matching structures.

In order to better utilize and protect marine organisms, reliable underwater object detection methods need to be developed. Due to various influencing factors from complex and changeable underwater environments, the underwater object detection is full of challenges. Therefore, this paper improves a two-stage algorithm of Faster RCNN (Regions with Convolutional Neural Network Feature) to detect holothurian, echinus, scallop, starfish and waterweeds. The improved algorithm has better performance in underwater object detection. Firstly, we improved the backbone network of the Faster RCNN, replacing the VGG16 (Visual Geometry Group Network 16) structure in the original feature extraction module with the Res2Net101 network to enhance the expressive ability of the receptive field of each network layer. Secondly, the OHEM (Online Hard Example Mining) algorithm is introduced to solve the imbalance problem of positive and negative samples of the bounding box. Thirdly, GIOU (Generalized Intersection Over Union) and Soft-NMS (Soft Non-Maximum Suppression) are used to optimize the regression mechanism of the bounding box. Finally, the improved Faster RCNN model is trained using a multi-scale training strategy to enhance the robustness of the model. Through ablation experiments based on the improved Faster RCNN model, each improved part is disassembled and then the experiments are carried out one by one, which can be known from the experimental results that, based on the improved Faster RCNN model, mAP@0.5 reaches 71.7%, which is 3.3% higher than the original Faster RCNN model, and the average accuracy reaches 43%, and the F1-score reaches 55.3%, a 2.5% improvement over the original Faster RCNN model, which shows that the proposed method in this paper is effective in underwater object detection.

This paper studies reconstruction of human body shape and pose from a single-view image. While most of current work attempts to regress parameters of human body model such as Skinned Multi-Person Linear Model (SMPL) and Hand Model with Articulated and Non-rigid Deformations (MANO), these parametric approaches underperform compared to non-parametric approaches. Due to the lack of the spatial relationship in the input image, the parametric approaches are hardly used to reconstruct the human body precisely. Besides, the rotation parameter regression is a complex task in parametric approaches. Therefore, we introduce a novel graph convolutional neural network (Graph CNN)-based framework for estimating a non-parametric mesh model. Our key innovation is that the proposed model is trained in a generative adversarial manner. Firstly, Graph CNN utilizes mesh topology to capture integral information of the full 3D human shape and then generate a more smooth and high-quality human mesh model. Secondly, the discriminator in our network acts as a supervisor to specify whether a human shape and pose are real or not. The generator is encouraged to generate human body mesh that is close to the manifold of the real human mesh distribution. Extensive experimental results demonstrate the effectiveness of our proposed framework. In contrast to the state-of-the-art methods, our method can achieve better performance in human shape and pose estimation.

To improve the accuracy of robotic grasp in some uncertain environments, a deep learning‐based object‐detection method for a five‐fingered industrial robot hand model is proposed in this study. The authors first design a five‐fingered industrial robot hand model with 21‐degrees of freedom (DOF). Based on the sensor data of a 5DT data glove, the industrial robot hand can be controlled in real time. They use the object‐detection network's faster regions convolutional neural network and single shot multibox detector to locate the grasp objects. To optimise the robotic grasp detection, two grasp‐predictor methods, direct grasp predictor and multi‐modal grasp predictor, are applied to obtain the best graspable region. In the simulation designed in this study, cooperating with a 6‐DOF robot arm, the five‐fingered industrial robot hand can detect an object accurately and grasp it steadily.

Joint 3D face reconstruction and dense alignment based on a single image has always been a challenge in computer vision. Recently, some work achieves these two goals simultaneously by adopting the position map to represent a 3D face. In this paper, we design a novel upsampling block, denoted as shortcut-upsampling block, to construct the position map. We first utilize the residual blocks to extract and downsample the feature maps from the 2D face image, and then utilize the proposed shortcut-upsampling blocks to convert the feature maps to the corresponding position map. Shortcut-upsampling block allows our model to be only 121M and achieve excellent performance on 3D face reconstruction and dense alignment. Compared with other deep learning methods based on 3D Morphable Model(3DMM), our model is the fastest when reconstructing a 3D face. In addition, through introducing a dynamic weight loss function, our model can converge the loss to a lower value and obtain a better performance on 3D face reconstruction.

Recent research works have highlight the invariance or the symmetry that exists in the weight space of a typical neural network and the negative effect of the symmetry on the training neural network due to the Euclidean gradient being not scaling-invariant. Although the problem of the symmetry can be solved by either defining a suitable Riemannian gradient, which is a scale-invariant or placing appropriate constraints on the weights, it will introduce very high-computation cost. In this paper, we first discuss various invariances or symmetries in the weight space, and then we propose to solve the problem via the scaling-invariance of the neural network itself, instead of the scaling-invariant updates methods. The motivation behind our method is that the optimized parameter point in the weight space may be moved from the ill-conditioning region to the flat region via a series of node-wise rescaling without changing the function represented by the neural network. Second, we proposed the scaling-based weight normalization. The proposed method can compatible with the commonly used optimization algorithms and collaborates well with batch normalization. Although our algorithm is very simple, it can accelerate the convergence speed. The additional computation cost introduced by our method is lower. Lastly, the experiments show that our proposed method can improve the performance of various networks architectures over large-scale datasets consistently. Our method outperforms the state-of-the-art methods on CIFAR-100: we obtain test errors as 17.18%.

Keyword spotting (KWS) is a major component of human-computer interaction for smart on-device terminals and service robots, the purpose of which is to maximize the detection accuracy while keeping footprint size small. In this paper, based on the powerful ability of DenseNet on extracting local feature-maps, we propose a new network architecture (DenseNet-BiLSTM) for KWS. In our DenseNet-BiLSTM, the DenseNet is primarily applied to obtain local features, while the BiLSTM is used to grab time series features. In general, the DenseNet is used in computer vision tasks, and it may corrupt contextual information for speech audios. In order to make DenseNet suitable for KWS, we propose a variant DenseNet, called DenseNet-Speech, which removes the pool on the time dimension in transition layers to preserve speech time series information. In addition, our DenseNet-Speech uses less dense blocks and filters to keep the model small, thereby reducing time consumption for mobile devices. The experimental results show that feature-maps from DenseNet-Speech maintain time series information well. Our method outperforms the state-of-the-art methods in terms of accuracy on Google Speech Commands dataset. DenseNet-BiLSTM is able to achieve the accuracy of 96.6% for the 20-commands recognition task with 223K trainable parameters.

Stereo images can provide more extra information to enhance performance of super-resolution (SR) owing to two different visual angles. Due to the attention mechanism has been widely employed in various computer vision tasks and achieved excellent performance, the recent methods for stereo SR utilize attention mechanism to obtain stereo correspondence. However, their attention mechanisms are generally spatial since they treat channel-wise features equally, which result in ignoring a lot of valuable information contained in the channels. In this paper, we propose a parallax-based spatial and channel attention stereo SR network (PSCASSRnet) to combine the stereo correspondence with the SR task. Specifically, we present a parallax-based spatial and channel attention module (PSCAM) to take full advantage of spatial and channel-wise characteristics to obtain the stereo correspondence. The extensive experiments prove that our PSCAM can produce more appropriate attention map with all possible disparities between the stereo image pair to achieve better SR performance, and our PSCASSRnet outperforms other state-of-the-art methods in the aspects of visual quality and quantitive on the three mainstream public datasets named KITTI2012, KITTI2015 and Middlebury respectively.

Reference-based image super-resolution (RefSR) methods have achieved performance superior to that of single image super-resolution (SISR) methods by transferring texture details from an additional high-resolution (HR) reference image to the low-resolution (LR) image. However, existing RefSR methods simply add or concatenate the transferred texture feature with the LR features, which cannot effectively fuse the information of these two independently extracted features. Therefore, this paper proposes a dual projection fusion for reference-based image super-resolution (DPFSR), which enables the network to focus more on the different information between feature sources through inter-residual projection operations, ensuring effective filling of detailed information in the LR feature. Moreover, this paper also proposes a novel backbone called the deep channel attention connection network (DCACN), which is capable of extracting valuable high-frequency components from the LR space to further facilitate the effectiveness of image reconstruction. Experimental results show that we achieve the best peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) performance compared with the state-of-the-art (SOTA) SISR and RefSR methods. Visual results demonstrate that the proposed method in this paper recovers more natural and realistic texture details.

Boundary equilibrium generative adversarial networks (BEGANs) can generate impressively realistic face images, but there is a trade-off between the quality and the diversity of generated images. Based on BEGANs, we propose an effective approach to generate images with higher quality and better diversity. By adding a second loss function (a denoising loss) to the discriminator, the discriminator can learn more useful information about the distribution of real images. Naturally, the ability of discriminator in distinguishing between real and generated images is improved, which further guides the generator to produce more realistic images to confuse the discriminator. We also find that using technique of batch normalization in BEGANs architecture can improve the diversity of generated images. By using batch normalization and adding a denoising loss to the objective of discriminator, we achieve comparative generations on CIFAR-10 and CelebA data sets. In addition, we evaluate the effect of several techniques on BEGANs framework through "Inception-Score", a measure which has been found to correlate well with human assessment of generated samples.

It is still a challenge for robots to interact with complex environments in a smooth and natural manner. The robot should be aware of its surroundings and inner status to make decisions accordingly and appropriately. Contexts benefit the interaction a lot, such as avoiding frequent interruptions (e.g., the explicit inputting requests) and thus are essential for interaction. Other challenges, such as shifting attentional focus to a more important stimulus, etc., are also crucial in interaction control. This paper presents a hybrid automatic control approach for interaction, as well as its integration, with these multiple important factors, aiming at performing natural, human-like interactions in robots. In particular, a novel approach of architectural attentional control, based on affection is presented, which attempts to shift the attentional focus in a natural manner. Context-aware computing is combined with interaction to endow the robot with proactive abilities. The long-term interaction control approaches are described. Emotion and personality are introduced into the interaction and their influence mechanism on interaction is explored. We implemented the proposal in an interactive head robot (IHR) and the experimental results indicate the effectiveness.

Although many researchers have begun to study the area of Cyber Physical Social Sensing (CPSS), few are focused on robotic sensors. We successfully utilize robots in CPSS, and propose a sensor trajectory planning method in this paper. Trajectory planning is a fundamental problem in mobile robotics. However, traditional methods are not suited for robotic sensors, because of their low efficiency, instability, and non-smooth-generated paths. This paper adopts an optimizing function to generate several intermediate points and regress these discrete points to a quintic polynomial which can output a smooth trajectory for the robotic sensor. Simulations demonstrate that our approach is robust and efficient, and can be well applied in the CPSS field.