Search

VLSI realization of video compression is the key to real-time high-definition 3D communication systems. The newly established multiview video coding standard, as an extension profile of H.264/AVC, draws more and more attention for its high compression ratio and free-viewpoint support. Besides providing the 3D experience, multiview video can also give users complete scene perception. However, the multiple-viewpoint throughput requirement of MVC increase the complexity and hardware cost dramatically. The system memory bandwidth, on-chip memory size, and processing data throughput of each module all need to be optimized in an MVC encoder. Therefore, efficient hardware solutions for MVC architecture design are needed. In this article an overview of 3D video coding standards developments and design challenges of an MVC encoder are discussed. Then the algorithm and architecture optimization schemes are proposed. For the trade-off between system memory bandwidth and on-chip memory size, a cache-based prediction engine is proposed to ease both design challenges. Moreover, the hybrid open-close loop intra prediction scheme and the frame-parallel pipeline-doubled dual CABAC solve the throughput requirement problem. At the end of this article, based on all the proposed solutions, a prototype single-chip MVC encoder design with processing ability of 4096 × 2160 single-view to 1280 × 720 seven-view is presented.

Multiview video coding (MVC) plays an important role in a 3-D video system. In addition, the resolution of HDTV is increasing to present more vivid perception for users. To realize real-time processing of dozens of TOPS, VLSI solution is necessary. However, ultra high computational complexity, a large amount of external memory bandwidth and on-chip SRAM size, and complex MVC prediction structures are three main design challenges of implementation of MVC hardware architecture. In this paper, an MVC single-chip encoder is proposed for H.264/AVC Multiview High Profile and High Profile for 3-D and quad full high definition (QFHD) TV applications, respectively. The 4096 × 2160 p multiview video encoder chip is implemented on a 11.46 mm2 die with 90 nm CMOS technology. An eight-stage macroblock pipelined architecture with proposed system scheduling and cache-based prediction core supports real-time processing from one-view 4096 × 2160 p to seven-view 720 p videos. The 212 Mpixels/s throughput is 3.4 to 7.7 times higher than previous work. The 407 Mpixels/W power efficiency is achieved, and 94% on-chip SRAM size and 79% external memory bandwidth are saved by the proposed techniques.

3-D video will be the most prominent video technology in the next generation. Among the 3-D video technologies, stereo video systems are considered to be realized first in the near future. Stereo video systems require double bandwidth and more than twice the computational complexity relative to mono-video systems. Thus, an efficient coding scheme is necessary for transmitting stereo video. In this paper, a new structure of prediction core in stereo video coding systems is proposed from the algorithm level to the hardware architecture level. The joint prediction algorithm (JPA), which combines three prediction schemes, is proposed for high coding efficiency and low computational complexity. It makes the system outperform MPEG-4 temporal scalability and simple profile by 2-3 dB in rate-distortion performance. Besides, JPA also utilizes the characteristics of stereo video and successfully reduces about 80% computational complexity. Then, a new hardware architecture of the prediction core based on JPA and a modified hierarchical search block-matching algorithm is proposed. With a special data flow, no bubble cycles exist during the block-matching process. The proposed architecture also adopts the near-overlapped candidates reuse scheme to save the heavy burden of data access. Besides, both on-chip memory requirement and off-chip memory bandwidth can be reduced by the proposed new scheduling. Compared with the hardware requirement for the implementation of full search block-matching algorithm, only 11.5% on-chip SRAM and 3.3% processing elements are needed with a tiny PSNR drop, making it area-efficient while maintaining high stereo video quality and processing capability

3DTV promises to become the mainstream of next-generation TV systems. Highresolution 3DTV provides users with a vivid watching experience. Moreover, free-viewpoint view synthesis (FVVS) extends the common two-view stereo 3D vision into virtual reality by generating unlimited views from any desired viewpoint. In the next-generation 3DTV systems, the set-top box (STB) SoC requires both a high-definition (HD) multiview video-coding (MVC) decoder to reconstruct the real camera-captured scenes and a free-viewpoint view synthesizer to generate the virtual scenes [1–2].

In recent years, 3D related applications, including 3D movie and 3DTV, are getting more and more attentions. In order to make the real-time 3D applications feasible, efficient multview video coding (MVC) scheme is desired. However, the system throughput requirement on a high definition (HD) MVC encoder is at least two times higher than the current highest throughput H.264/AVC encoder design. In order to meet the target HD MVC specifications, the system scheduling analysis for the HD MVC encoder is performed in this paper. By the 8-stage MB pipeline system architecture with view-parallel MB-interleaved system scheduling and other in-core computation optimizations, the proposed HD MVC encoder design provides about 339% to 1536% higher system throughput than the previous HD H.264/AVC encoder chips[4][5][6] and therefore the real-time HD MVC encoding can be achieved.

Fractional motion estimation (FME) is widely used in video compression standards. In H.264/AVC, the precision of motion vector is down to quarter pixels to improve the coding efficiency. However, FME occupies over 45% of the computation complexity in an H.264 encoder and this high complexity limits the processing capability. In this paper, a single-iteration full search FME is proposed. By the algorithm and architecture co-optimization, the bandwidth to the frame buffer is reduced by 31%. Furthermore, 82% of circuit area for the Hadamard transformation and subtraction are saved from the direct implementation. Compared with prior arts, the proposed design supports 3.39 × higher throughput with only 0.02 dB PSNR drop. Thus, the specification of 4096 × 2160 quad full high definition H.264/AVC FME processing can be achieved.

Multiview video can bring the 3D and virtual reality perceptual experience to users by its multiple view point characteristic. To support the smooth and free view point switching, both matrix-based depth image based rendering and the complex virtual view interpolation schemes are required. In order to reduce the high computation complexity and avoid the iterative processing scheduling in conventional view interpolation flows, a single iteration view interpolation algorithm is proposed. By use of the proposed algorithm, the redundant warping operations are reduced by 86%. In addition, based on the proposed artifact detecting and removing algorithm, artifacts due to imperfect depth maps can be detected and eliminated at the same time. Therefore, no additional post-processing or iteration is required and the single iteration processing is achieved.

To provide more vivid perception, more and more advanced features, like the 4k×2k resolution and the multiview functionality, are emerging for TV. For a multiview video coding (MVC) encoder, motion and disparity estimation (ME/DE) take at least half the hardware requirement. To solve these challenges, a cache-based integer ME/DE algorithm is proposed. With a cache memory as the search window buffer, a predictor-centered ME/DE algorithm is presented. The search range can be reduced to ±16 pixels with less than 0.1dB quality drop compared with full search algorithm. Based on this algorithm, an integer ME/DE chip design is realized. It can reduce 82% on-chip SRAM and 39% system bandwidth. Moreover, the search candidate requirement is also reduced by 79%. As the result, an ME/DE chip design for 4k×2k quad-HD H.264 and HDTV MVC is implemented.

Multiview video coding (MVC) is getting more and more attention in recent years because of the realistic perception it can bring. However, the current H.264-based coding scheme cannot get enough coding gain in the rate-distortion (R-D) performance. Furthermore, the view synthesis and random access between views are also design challenges. In this paper, a multiview hybrid coding system with texture-depth synthesis is proposed. By sharing the prediction information between texture and depth maps, the depth map can be reconstructed with reduced bitrate overhead. Then, after using matrix-based warping and adaptive disparity compensation, the R-D performance of the proposed system can be about 2 to 4 dB higher than the current MVC coding scheme at low bitrate range.

This paper presented a novel depth map generation method - the short-term motion assisted color segmentation, which combines the pictorial, monocular and binocular depth cues of human vision. The proposed method utilizes a motion/edge registration technique to avoid the motion jitter error in common motion segmentation. And the motion/image segment adaptation algorithm matches the connected components with the motion segments. Even for static scene, the connected component algorithm is still working for the depth map generation. The experimental results show that the adaptation of motion and image segmentation improves quality and smoothness of the depth map both in the spatial and temporal domain.

Multiview video coding (MVC) plays a critical role in 3D-TV systems. However, MVC require much more memory bandwidth and computational complexity relative to mono-view video coding. Therefore, an efficient prediction scheme is necessary for computation reduction. In this paper, a new fast prediction algorithm, computation-free motion estimation (ME) with inter-view mode decision, is proposed. By utilizing disparity estimation to find corresponding blocks between different views, the coding information, such as modes and motion vectors, can be effectively reused from the coded view channel. Therefore, the computation for ME can be almost totally neglected in most view channels. Besides, the proposed algorithm also provides the computation scalability. Experimental results show that compared with the full search block matching algorithm (FSBMA), the proposed algorithm not only save 61-72% computation with near-FSBMA quality, but save almost 100% computation with a little quality drop of 0.23-0.36 dB. Thus, computation-free ME is achieved.

Multiview video coding (MVC) systems require much more bandwidth and computational complexity relative to mono-view video systems. Thus, when designing a VLSI architecture for MVC systems, the hardware resource allocation is a critical issue. In this paper, we propose a new system bandwidth analysis scheme for various and complicated MVC structures. The precedence constraint in the graph theory is adopted for deriving the processing order of frames in a MVC system. In addition, current block centric scheduling (CBCS) and search window centric scheduling (SWCS) are proposed for MVC bandwidth analysis. By adopting data reuse schemes, several design points are explored with the aid of the proposed analysis scheme. The suitable hardware resource allocation can be easily determined.

Luminance and chrominance correction (LCC) is important in multi-view video coding (MVC) because it provides better rate-distortion performance when encoding video sequences captured by ill-calibrated multi-view cameras. This paper presents a robust and fast LCC algorithm based on motion compensated linear regression which reuses the motion information from the encoder. We adopt the linear weighted prediction model in H.264/AVC as our LCC model. In our experiments, the proposed LCC algorithm outperforms basic histogram matching method up to 0.4dB with only few computational overhead and zero external memory bandwidth. So, the dataflow of this method is suitable for low bandwidth/low power VLSI design for future multi-view applications.

3D video will become noticeable video technology in the next generation. In this paper, a stereo video coding system is proposed from algorithm level to hardware architecture level. We propose a novel stereo video coding system by exploiting joint block compensation scheme to achieve high coding efficiency. It is also suitable for hardware implementation. Due to more than twice computational complexity relative to mono video coding systems, a new hardware architecture based on hierarchical search block matching algorithm (HSBMA) with some modification is proposed. With special data flow, no bubble cycles exist during block matching process. Proposed architecture also adopts near overlapped candidates reuse scheme (NOCRS) to save heavy burden of data access. Besides, by the proposed new scheduling, both on-chip memory requirement and offchip memory bandwidth can be reduced. A prototype chip can achieve real-time requirement under the operating frequency of 81 MHz for 30 D1 frames per second (fps) in left and right channel simultaneously, with ME/DE search range of [-64, +63] in horizontal direction and [-32, +31]/[-16, +15] in vertical direction. Compared with the hardware requirement for implementation of full search block matching algorithm (FSBMA), only 11.5% on-chip SRAM and 1/30 amount of PEs are needed. It shows that the hardware cost is quite small.

Stereo video systems require double bandwidth and more than twice the computational complexity relative to mono-video systems. Thus, an efficient coding scheme is necessary for transmitting stereo video. We propose a novel stereo video coding system by exploiting a joint prediction scheme which combines three prediction schemes to achieve high coding efficiency and low computational complexity. Joint block compensation improves the visual quality. Motion vector prediction and mode pre-decision utilize the features of stereo video to reduce the computational complexity with up to 8-9 times acceleration. Experiments show that the proposed joint prediction scheme is 2 dB better than MPEG-4 TS and 3 dB better than MPEG-4 SP.

An efficient coding system is required for immersive teleconference to transmit stereo video. We propose a novel stereo video coding system by exploiting mesh-based disparity estimation and compensation scheme to achieve high coding efficiency and view synthesis ability. Based on the base-layer-enhance-layer structure, this system can provide stereoscopic scalability and compatibility to standards. With considering asymmetric spatial resolution property, good subjective quality can be achieved in ultra low bitrate situation. A novel fast disparity estimation algorithm named as two-stage iterative block and octagonal matching (TS-IBOM) algorithm is also proposed for this system. Experiments show that the proposed disparity estimation algorithm can generate accurate disparity vectors quickly. It is also shown that the proposed coding system has better coding efficiency than MPEG-4 simple profile and temporal scalability tool. The ultra low bitrate of 69 Kbps can be reached to encode 384/spl times/192 60 fps stereo video.