Enhance the 3D Object Detection With 2D Prior

In the modern field of multi-view 3D detection, interest has seen a notable rise. Existing methods are predominantly focused on constructing dense BEV (Bird’s Eye View) features or utilizing sparse queries for detection. In this paper, we synergize the methodologies of sparse queries with dense BEV feature representation and integrate robust 2D detection capabilities to propose an innovative structure. Specifically, we leverage the results of 2D detection and their predicted depth to generate a series of sparse queries endowed with strong semantic and positional priors. These queries demonstrate enhanced capability in capturing challenging cases in 3D detection. Additionally, we employ 2D detection and depth information to discern critical foreground BEV queries. This strategy enables a efficient and noiseless feature aggregation, leveraging multi-view image features. Our methodology extends to a comprehensive global modeling of all BEV queries, ensuring the derivation of premium quality BEV features. Furthermore, we enhance BEV features through temporal modeling that differentiates between static and dynamic objects. Our proposed approach achieves state-of-the-art results on the nuScenes test set with 60.4%mAP and 51.7%NDS. Code will be available.