Learning a convolutional neural network for fractional interpolation in HEVC inter coding

Motion compensated prediction (MCP) is an effective technology for video coding to improve compression efficiency. Fractional sample precision prediction is utilized in HEVC to further remove temporal redundancy, and finite impulse response (FIR) filters designed using decomposition of the discrete cosine transform are applied to generate samples that do not fall on the integer positions. However, the coefficients of these DCT-based interpolation filters are fixed, which may not be able to adapt to varied video content. Inspired by the remarkable success of convolutional neural network (CNN) in the single image super-resolution task, we propose to learn a convolutional neural network for fractional interpolation in HEVC inter prediction. Compared with super-resolution, there is one big difference in fractional interpolation — fractional interpolation needs to maintain samples at integer positions while super-resolution generates a whole high-resolution image. Another difference is no real ground truth is available in fractional interpolation process. To overcome these two challenges, we introduce a constraint strategy to the training phase of the original super-resolution network as well as a specially designed preprocessing step which reuses the DCTIF interpolation process. Unlike other previous work, our proposed approach simultaneously generating the fractional positions from one network and experimental results show our proposed approach achieves 0.45% BD-Rate reduction under the low-delay-P configuration on average.