Search

3D object detection is an essential component of scene perception and motion prediction in autonomous driving. Previous methods represent objects as the truncated signed distance fields (3D bounding box), which can only provide the geometric constraints of point-to-line. In this work, we define the object as a more compact representation, quadric (ellipsoid) in a 3D scene and a conic (ellipse) in an image, which can provide stronger geometric constraints of surface-to-curve. Specifically, we estimate a ellipsoid from a conic fitted by a 2D bounding box to obtain 3D object localization and occupancy. We further to formulate this constraint relation as a nonlinear optimization problem in dual space, which enables us to easy recover stable and accurate 3D object parameters by adding only three additional direction-aware branches to the existing 2D detection networks. In addition, we decouple the dimensions of object and update the length and orientation of objects in our iterative algorithm when the estimations from the 2D detection networks have different deviations. The final detection results can be obtained after passing through our geometry-related refinement network. We evaluate our method on the KITTI object detection benchmark and achieve the best performance among published monocular competitors. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel deep architecture by combining multiple sensors for 3D object detection, named MSL3D. While recently LiDAR-Camera methods introduce additional semantic cues, working with fewer false detections, there is still a performance gap compared LiDAR-only methods. We argue that this gap is caused for two reasons: 1) the 3D spherical receptive fields of the set abstraction of the point clouds are not aligned with the 2D pixel-level receptive fields of the image. 2) the premature introduction of image information makes it is difficult to apply data augmentation both LiDAR and image synchronously. For the first problem, we extend 3D set abstraction to a 2D set abstraction that can transform the 2D image features to the 3D sphere to unify the receptive field of multi-modal data. For the second problem, we design a novel two-stage 3D detection framework that employs the LiDAR-only backbone in the first stage to estimate high-recall and high-quality proposals and then integrates the image and point clouds information for box refinement and confidence prediction. Besides, we add two auxiliary networks to effectively learn image features and point cloud features when using different multi-modal data augmentation strategies synchronously. Moreover, we design a consistency-structure generator using stereo images to determine whether any of a point in the 3D space belongs to the contour of the object, thereby supplementing the sparse point cloud information. Extensive experiments on the popular KITTI 3D objects detection dataset show that our proposed MSL3D achieves better performance comparing with other LiDAR-Only or LiDAR-Camera fusion approaches. [ABSTRACT FROM AUTHOR]

As two major data modalities in autonomous driving, LiDAR point clouds and RGB images include rich geometric clues and semantic features. Compared with using a single data modality, fusing two data modalities can provide complementary information for the 3D object detection task. However, some prevalent cross-modal methods (Vora et al., 2020; Huang et al., 2020; Sindagi et al., 2019) cannot effectively obtain favorable information, and only adopt a unilateral fusion mechanism. In this paper, we propose a novel fusion strategy named Bilateral Content Awareness Fusion (BCAF) to address these issues. Specifically, BCAF adopts a two-stream structure consisting of a LiDAR Content Awareness (LCA) branch and an Image Content Awareness (ICA) branch along with a Soft Fusion (SF) module. First, the LCA and ICA are used to enhance instance-relevant clues. Then, with two awareness features given by the LCA and ICA branches, the aggregation features can be generated to choose favorable image features and LiDAR features. Finally, the SF module fuses the bilateral favorable features and outputs the cross-modal feature. Experiments of our method are conducted on the KITTI dataset, including 3D object detection evaluation and bird's eye view evaluation. Compared with the previous art method, our approach achieves significant improvements. Especially for the metric of mean Average Precision (mAP) on the C a r category, our approach obtains 0.5 and 0.62 gains for the tasks of 3D object detection and bird's eye view, respectively. [ABSTRACT FROM AUTHOR]

Recently, the progress in autonomous driving tries to leverage the strong complementarity of LiDAR point clouds and RGB images to realize a high-efficient 3D object detection task. However, some works just simply decorate the raw point clouds or point-cloud features with camera clues in a hard way, which cannot fully exploit the relevance between the two-modal data. In this paper, we propose a dual-feature interaction module that adopts a soft-fusion strategy to give guidance for the LiDAR-camera feature fusion by interacting the LiDAR and camera features with Transformer. Compared with the hard-fusion method, this soft-fusion method can decorate the LiDAR feature with a reliable image feature. Additionally, we design an uncertainty-based 3D Intersection over Union (IoU) metric in the training process. This strategy aims at modeling the unreliability of 3D IoU scores to alleviate the bad effects caused by the coupling problem of 3D properties. Experiments conducted on the KITTI dataset achieve significant improvements in the 3D object detection and bird's eye view tasks when compared with the previous arts. Especially for the task of 3D object detection, our approach obtains 0.68 and 0.45 gains for the metric of A P 3 D on the moderate level and hard level, respectively. [ABSTRACT FROM AUTHOR]

Currently, the primary focus of 3D point-based detectors is to enhance their performance by primarily increasing the recall rate of foreground points at the semantic level. However, this approach is suboptimal. On one hand, ground-truth annotations of 3D bounding boxes are inherently ambiguous due to occlusions, signal distortion, or manual annotation errors, which can obscure genuine semantic information. On the other hand, the saliency of each point varies. In this paper, we propose SPSNet based on Bayes theorem to explore the mapping relationship between the sampled point and the corresponding bounding box, which we call point stability. This is a plug-and-play module and used as the basis for downsampling instead of relying on semantic information obtained through hard supervision of 3D bounding boxes. We incorporate the proposed method into the most popular point-based detector, IA-SSD, called SPSNet-IA. On the challenging KITTI validation set, with only changes to the downsampling strategy, it obtains 1.18, 2.78 and 2.96 gains for the metric of A P 3 D on the easy, moderate and hard levels, respectively. In addition, SPSNet-IA outperforms all published point-based approaches by a large margin and ranks 1st among single-modal methods. We also design qualitative and quantitative experiments to explain the meaning of stability learned from SPSNet. These additional experiments demonstrate that SPSNet is not only a performance-enhancing model but also a saliency analysis model. The code is available at https://github.com/AlanLiangC/SPSNet.git. • A valuable stability-aware downsampling strategy is proposed. • We define the concept of stability and propose SPSNet to obtain it. • We design sufficient experiments to demonstrate the role of SPSNet. • We explore the Predictive Power of SPSNet. [ABSTRACT FROM AUTHOR]