Search

This paper introduces architecture with high throughput, low on-chip memory, and efficient data access for Improved Dense Trajectories (iDT) as video representations for realtime action recognition. The iDT feature can capture longterm motion cues better than any existing deep feature, which makes it crucial in state-of-the-art action recognition systems. There are three major features in our architecture design, including a low bandwidth frame-wise feature extraction, low on-chip memory architecture for point tracking, and two-stage trajectory pruning architecture for low bandwidth. Using TSMC 40nm technology, our chip area is 3.1 mm2, and the size of on-chip memory is 40.8 kB. The chip can support videos in resolution of 320×240 with a throughput of 203 fps under 215 MHz, which is a 81.2 times speedup compared with CPU. Under the same operating frequency, it can also provide feature extraction for six windows of size 320 × 240 in higher resolution videos with a throughput of 34 fps.

In this paper, we propose a compact hardware architecture which can support HOG, HOF, MBHx and MBHy feature extraction in real-time. To achieve low-cost hardware, we first design a hardware-oriented algorithm. In addition to simplification of complicated numeric computation, we also modify the computation of the gradient at boundary to reduce on-chip buffers. The architecture is highly parallel in order to fully reuse input data and to achieve high throughput. Other features of our architecture include a four-bank memory composed of four SRAMs, a 2-cycle weighted binning, and a 2-port memory for histogram collection. The final specification can support HOG, HOF, MBHx, and MBHy feature extraction with a throughput of 65 fps in full-HD videos operating at 215 MHz, consuming only 57.6 kB on-chip memory and a gate count of 411k.

In this paper, we propose an accelerated local feature extraction in a reuse scheme for action recognition. Most local features of the previous frame could be reused due to the high correlation between successive frames. Feature extraction is only needed to be applied partially in the current frame. The full-frame features of the current frame are combined by features extracted at different times. Experimental results show that the proposed reuse scheme can achieve comparable performance to full-frame feature extraction while computational effort is reduced significantly.