1. Learning Super-Resolution Reconstruction for High Temporal Resolution Spike Stream
- Author
-
Yixuan Wang, Yonghong Tian, Xiang Xijie, Tiejun Huang, Jianing Li, and Lin Zhu
- Subjects
Quantitative Biology::Neurons and Cognition ,Pixel ,business.industry ,Computer science ,Feature extraction ,ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION ,Optical flow ,Iterative reconstruction ,Computer Science::Emerging Technologies ,Computer Science::Computer Vision and Pattern Recognition ,Media Technology ,Spike (software development) ,Computer vision ,Artificial intelligence ,Electrical and Electronic Engineering ,business ,Projection (set theory) ,Encoder ,Image resolution - Abstract
Spike camera is a new type of bio-inspired vision sensor, each pixel of which perceives the brightness of the scene independently, and finally outputs 3-dimensional spatiotemporal spike streams. To bridge the spike camera and traditional frame-based vision, there is some works to reconstruct spike streams into regular images. However, the low spatial resolution (400×250) of the spike camera limits the quality of the reconstructed images. Thus, it is meaningful to explore a super-resolution reconstruction for spike streams. In this paper, we propose an end-to-end network to reconstruct high-resolution images from low-resolution spike streams. To utilize more spatiotemporal features of spike streams, our network adopts a multi-level features learning mechanism, including intra-stream feature extraction by spike encoder, inter-stream dependencies extraction based on optical flow module, and joint features learning via spike-based iterative projection. Experimental results demonstrate that our network is superior to the combination of state-of-the-art intensity image reconstruction methods and super-resolution networks on simulated and real datasets.
- Published
- 2023