Search

One of the problems that still adversely affect the optical flow accuracy is the missing parts or the loss of some details in the flow map that is estimated by the flow estimation network. The main reasons are the feature maps that are extracted by the local kernels of the convolution neural networks (CNN), which cause feature loss and provide low-quality features, and the occluded parts after the image reconstruction. This loss in the encoder part becomes hard to recover in the flow estimation network. To overcome this problem, we propose a CNN with transformer architecture to emphasize the important features and model the long-range dependencies to produce a better feature representation that can provide the flow map with strong details. In addition, we propose an occlusion compensation loss to rectify the occlusion map and assist the network in learning how to predict the flow in the occluded regions. Extensive experiments are conducted on Sintel and KITTI benchmarks, and the results demonstrate the efficiency of our model in increasing the accuracy of the optical flow and improving the flow in the occluded regions. [ABSTRACT FROM AUTHOR]

Scene flow estimation is one of the most crucial components of many scene understanding tasks, which represents the complete 3D motion of objects in dynamic scene. Most of existing scene flow estimation approaches are usually based on joint learning framework, which cast the problem of scene flow estimation to dense prediction of optical flow and stereo matching. However, these approaches need to reconstruct 3D motion from optical flow and disparity using 2D stereo image pairs, which leads the estimation process to be indirect. Recently, FlowNet3D attempts to learn scene flow from 3D point clouds, which adopts element-wise max-pooling to aggregate features from different points. Nevertheless, max-pooling operation only can obtain the strongest activation on features across a local or global region, which may increase the loss of some useful detailed and contextual. Specifically, for dense estimation task, the ability to transfer information gradually from coarse to fine layers is important. To address this problem, we investigate a new deep architecture, 3D point convolutional network, to learn scene flow from 3D point clouds. This specific architecture uses multi-layer perceptron (MLP) to approximate the weight function and applies a density scale to re-weight the learned weight functions for each convolutional filter, which can make the network permutation-invariant and translation invariant on 3D space which is beneficial for feature aggregation. Extensive experimental results are conducted on FlyingThings3D and KITTI scene flow datasets and show the effectiveness of our proposed approach. Our algorithm can achieve state-of-the-art performance on FlyingThings3D and KITTI datasets. [ABSTRACT FROM AUTHOR]

Recently, optical flow estimation benefits greatly from deep learning based techniques. Most approaches use encoder-decoder architecture (U-Net) or spatial pyramid network (SPN) to learn optical flow. Both U-Net and SPN can extract multi-scale features and can predict optical flow directly. However, existing networks ignore to exploit the global information among channel features and inter-spatial relationship of features. In this paper, we propose a dual self-attention pyramid network, which adaptively integrates local features with their global dependencies and focuses on important features and suppresses unimportant features. Specifically, we introduce two types of attention modules into SPN, which emphasizes meaningful features along channel and spatial axes. The channel attention can adaptively re-weight channel-wise features by considering interdependencies among channels. Moreover, the spatial attention can utilize global contextual information to emphasize or suppress features in different spatial locations. In addition, two attention modules are embedded into each pyramidal level, which can refine features at different scale. We evaluate our method on MPI-Sintel and KITTI. The experimental results show that using the dual self-attention module can improve the representation power of network and further increase the accuracy of optical flow estimation. [ABSTRACT FROM AUTHOR]

Supervised depth prediction and optical flow estimation have achieved promising performance due to the advanced deep network architectures. Since the ground truths are difficult to be collected, many recent works try to learn the depth and flow in an unsupervised manner. However, existing methods only use features from convolutional layers or a simple aggregation of multi-level features to predict the depth and flow maps, which is insufficient to exploit context information. In this paper, we attempt to exploit object contextual information and investigate the effect of the object context for joint learning of depth and optical flow. Specifically, we present a novel combination of object context and the framework of joint learning depth and optical flow. Our proposed network can exploit and integrate the object context for both tasks by aggregating the context according to pair-wise similarities. Furthermore, we adopt the existing spatial pyramid network (SPN) to estimate the depth and flow in a coarse-to-fine strategy effectively. Given temporally adjacent stereo pairs, our network can be trained end-to-end in an unsupervised manner and can predict the depth and flow maps simultaneously. We conduct experiments on two publicly available datasets, KITTI2012 and KITTI2015. Our proposed approach yields comparable performance on both depth and flow tasks, compared to the recent deep learning-based approaches. Experimental results demonstrate that exploiting object contextual information is useful and beneficial for depth and optical flow estimation. [ABSTRACT FROM AUTHOR]

The convolution neural network is still the main tool for extracting the image features and the motion features for most of the optical flow models. The convolution neural networks cannot model the long-range dependencies, and more details are lost in deeper layers. All the deficiencies in the extracted features affect the estimated flow. Therefore, in this work, we concentrated on optimizing the convolution neural network in both the encoder and decoder parts to improve the image and motion features. To enhance the image features, we utilize the involution to provide rich features and model the long-range dependencies. In addition, we propose a Multi-Scale-Interaction module which utilizes the self-attention to make an interaction between the feature scales to avoid detail loss. Additionally, we propose a Motion-Features-Optimization block that utilizes the deformable convolution to enhance the motion features. Our model achieves the state-of-the-art performance on Sintel and KITTI 2015 benchmarks. • We utilize the involution operation in our unsupervised optical flow model to improve pixel-wise matching and, accordingly, the estimated flow. • We propose a multi-scale-interaction module to prevent detail loss at each pyramid scale. • We propose a motion-feature-optimization block to improve the motion features. [ABSTRACT FROM AUTHOR]

• We propose a feature masking method to handle the occlusion in optical flow and depth estimation. • We propose a deformable decoder to improve the structural details in occluded and non-occluded regions of optical flow and depth. • We propose an Update-flow block to refine the optical flow. In this work, we improve the optical flow and depth based on the important observation that they should share the geometric structure of the reference image. We initially propose a feature masking method to reduce the occlusion impact on the optical flow and depth by producing preliminary motion features that share the structure of the reference image. In addition, we propose a deformable decoder that learns the geometrical structure of the reference image in the form of a group of offsets and uses them to adapt the motion features of flow and depth maps, thereby preventing incorrect propagation in occluded regions and providing more structural details in the other regions. Furthermore, we recursively update the optical flow with self-supervised cues learned from the rigid flow and optical flow. Our method achieves a new state-of-the-art result for the optical flow on the KITTI 2015 benchmark with F1 = 11.17%. [ABSTRACT FROM AUTHOR]

• To the best of our knowledge, this is the first survey paper to cover both optical flow and scene flow estimation comprehensively. • In contrast to existing surveys, this survey includes knowledge-driven, data-driven and hybrid-driven methods. • Comprehensive comparisons of existing methods on several datasets are provided with insightful observations and sufficient analyses. • Some of the open issues and potential research directions are discussed. Motion analysis is one of the most fundamental and challenging problems in the field of computer vision, which can be widely applied in many areas, such as autonomous driving, action recognition, scene understanding, and robotics. In general, the displacement field between subsequent frames can be divided into two types: optical flow and scene flow. The optical flow represents the pixel motion of adjacent frames. In contrast, the scene flow is a 3D motion field of the dynamic scene between two frames. Traditional approaches for the estimation of optical flow and scene flow usually leverage the variational technique, which can be solved as an energy minimization process. In recent years, deep learning has emerged as a powerful technique for learning feature representations directly from data. It has led to remarkable progress in the field of optical flow and scene flow estimation. In this paper, we provide a comprehensive survey of optical flow and scene flow estimation. First, we briefly review the pioneering approaches that use variational technique and then we delve in detail into the deep learning-based approaches. Furthermore, we present insightful observations on evaluation issues, specifically benchmark datasets, evaluation metrics, and state-of-the-art performance. Finally, we give the promising directions for future research. To the best of our knowledge, we are the first to review both optical flow and scene flow estimation, and the first to cover both traditional and deep learning-based approaches. [ABSTRACT FROM AUTHOR]

Learning optical flow based on convolutional neural networks has made great progress in recent years. These approaches usually design an encoder-decoder network that can be trained end-to-end. In encoder part, high-level feature information is extracted through a series of strided convolution, which is similar to most image classification networks. In contrast to classification task, spatial feature maps are then enlarged to full scale of input by conducting successive deconvolution layer in decoder part. However, optical flow estimation is a pixel-level task, and blurry flow fields are usually generated, which is caused by unrefined features and low-resolution. To address this problem, we propose a novel network, which combines attention mechanism and dilated convolutional neural network. In this network, the channel-wise features are adaptively weighted by building interdependencies among channels, which can weaken the weights of useless features and can enhance the directivity of feature extraction. Meanwhile, spatial precision is achieved by employing dilated convolution which improves the receptive field without large computational source and keeps the spatial resolution of feature map unchanged. Our network is trained on FlyingChairs and FlyingThings3D datasets in a supervised manner. Extensive experiments are conducted on MPI-Sintel and KITTI datasets to verify the effectiveness of the proposed method. The experimental results show that attention mechanism and dilated convolution are beneficial for optical flow estimation. Moreover, our method achieves better accuracy and visual improvements comparing to most of recent approaches. [ABSTRACT FROM AUTHOR]