Back to Search
Start Over
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation
- Source :
- CVPR
- Publication Year :
- 2017
- Publisher :
- IEEE, 2017.
-
Abstract
- Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video super-resolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is end-to-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to single-frame models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-the-art performance in both accuracy and efficiency.<br />Comment: Changes: * Uploaded Vid4 results (footnote 1). * Added references [14, 29] as spatial-transformer prior art. * Fixed typos
- Subjects :
- FOS: Computer and information sciences
Image fusion
Motion compensation
Artificial neural network
Computer science
business.industry
Computer Vision and Pattern Recognition (cs.CV)
Computer Science - Computer Vision and Pattern Recognition
ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION
020206 networking & telecommunications
02 engineering and technology
Iterative reconstruction
Convolutional neural network
Convolution
0202 electrical engineering, electronic engineering, information engineering
020201 artificial intelligence & image processing
Computer vision
Artificial intelligence
business
Image resolution
Subjects
Details
- Database :
- OpenAIRE
- Journal :
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Accession number :
- edsair.doi.dedup.....5e8a31886b0def939f3b6b78edab3226