Search

VLSI realization of video compression is the key to real-time high-definition 3D communication systems. The newly established multiview video coding standard, as an extension profile of H.264/AVC, draws more and more attention for its high compression ratio and free-viewpoint support. Besides providing the 3D experience, multiview video can also give users complete scene perception. However, the multiple-viewpoint throughput requirement of MVC increase the complexity and hardware cost dramatically. The system memory bandwidth, on-chip memory size, and processing data throughput of each module all need to be optimized in an MVC encoder. Therefore, efficient hardware solutions for MVC architecture design are needed. In this article an overview of 3D video coding standards developments and design challenges of an MVC encoder are discussed. Then the algorithm and architecture optimization schemes are proposed. For the trade-off between system memory bandwidth and on-chip memory size, a cache-based prediction engine is proposed to ease both design challenges. Moreover, the hybrid open-close loop intra prediction scheme and the frame-parallel pipeline-doubled dual CABAC solve the throughput requirement problem. At the end of this article, based on all the proposed solutions, a prototype single-chip MVC encoder design with processing ability of 4096 × 2160 single-view to 1280 × 720 seven-view is presented.

Multiview video coding (MVC) plays an important role in a 3-D video system. In addition, the resolution of HDTV is increasing to present more vivid perception for users. To realize real-time processing of dozens of TOPS, VLSI solution is necessary. However, ultra high computational complexity, a large amount of external memory bandwidth and on-chip SRAM size, and complex MVC prediction structures are three main design challenges of implementation of MVC hardware architecture. In this paper, an MVC single-chip encoder is proposed for H.264/AVC Multiview High Profile and High Profile for 3-D and quad full high definition (QFHD) TV applications, respectively. The 4096 × 2160 p multiview video encoder chip is implemented on a 11.46 mm2 die with 90 nm CMOS technology. An eight-stage macroblock pipelined architecture with proposed system scheduling and cache-based prediction core supports real-time processing from one-view 4096 × 2160 p to seven-view 720 p videos. The 212 Mpixels/s throughput is 3.4 to 7.7 times higher than previous work. The 407 Mpixels/W power efficiency is achieved, and 94% on-chip SRAM size and 79% external memory bandwidth are saved by the proposed techniques.

3-D video will become one of the most significant video technologies in the next-generation television. Due to the ultra high data bandwidth requirement for 3-D video, effective compression technology becomes an essential part in the infrastructure. Thus multiview video coding (MVC) plays a critical role. However, MVC systems require much more memory bandwidth and computational complexity relative to mono-view video coding systems. Therefore, an efficient prediction scheme is necessary for encoding. In this paper, a new fast prediction algorithm, content-aware prediction algorithm (CAPA) with inter-view mode decision, is proposed. By utilizing disparity estimation (DE) to find corresponding blocks between different views, the coding information, such as rate-distortion cost, coding modes, and motion vectors, can be effectively shared and reused from the coded view channel. Therefore, the computation for motion estimation (ME) in most view channels can be greatly reduced. Experimental results show that compared with the full search block matching algorithm (FSBMA) applied to both ME and DE, the proposed algorithm saves 98.4-99.1% computational complexity of ME in most view channels with negligible quality loss of only 0.03-0.06 dB in PSNR.

3-D video will be the most prominent video technology in the next generation. Among the 3-D video technologies, stereo video systems are considered to be realized first in the near future. Stereo video systems require double bandwidth and more than twice the computational complexity relative to mono-video systems. Thus, an efficient coding scheme is necessary for transmitting stereo video. In this paper, a new structure of prediction core in stereo video coding systems is proposed from the algorithm level to the hardware architecture level. The joint prediction algorithm (JPA), which combines three prediction schemes, is proposed for high coding efficiency and low computational complexity. It makes the system outperform MPEG-4 temporal scalability and simple profile by 2-3 dB in rate-distortion performance. Besides, JPA also utilizes the characteristics of stereo video and successfully reduces about 80% computational complexity. Then, a new hardware architecture of the prediction core based on JPA and a modified hierarchical search block-matching algorithm is proposed. With a special data flow, no bubble cycles exist during the block-matching process. The proposed architecture also adopts the near-overlapped candidates reuse scheme to save the heavy burden of data access. Besides, both on-chip memory requirement and off-chip memory bandwidth can be reduced by the proposed new scheduling. Compared with the hardware requirement for the implementation of full search block-matching algorithm, only 11.5% on-chip SRAM and 3.3% processing elements are needed with a tiny PSNR drop, making it area-efficient while maintaining high stereo video quality and processing capability

3DTV promises to become the mainstream of next-generation TV systems. Highresolution 3DTV provides users with a vivid watching experience. Moreover, free-viewpoint view synthesis (FVVS) extends the common two-view stereo 3D vision into virtual reality by generating unlimited views from any desired viewpoint. In the next-generation 3DTV systems, the set-top box (STB) SoC requires both a high-definition (HD) multiview video-coding (MVC) decoder to reconstruct the real camera-captured scenes and a free-viewpoint view synthesizer to generate the virtual scenes [1–2].

In recent years, 3D related applications, including 3D movie and 3DTV, are getting more and more attentions. In order to make the real-time 3D applications feasible, efficient multview video coding (MVC) scheme is desired. However, the system throughput requirement on a high definition (HD) MVC encoder is at least two times higher than the current highest throughput H.264/AVC encoder design. In order to meet the target HD MVC specifications, the system scheduling analysis for the HD MVC encoder is performed in this paper. By the 8-stage MB pipeline system architecture with view-parallel MB-interleaved system scheduling and other in-core computation optimizations, the proposed HD MVC encoder design provides about 339% to 1536% higher system throughput than the previous HD H.264/AVC encoder chips[4][5][6] and therefore the real-time HD MVC encoding can be achieved.

Multiview video can bring the 3D and virtual reality perceptual experience to users by its multiple view point characteristic. To support the smooth and free view point switching, both matrix-based depth image based rendering and the complex virtual view interpolation schemes are required. In order to reduce the high computation complexity and avoid the iterative processing scheduling in conventional view interpolation flows, a single iteration view interpolation algorithm is proposed. By use of the proposed algorithm, the redundant warping operations are reduced by 86%. In addition, based on the proposed artifact detecting and removing algorithm, artifacts due to imperfect depth maps can be detected and eliminated at the same time. Therefore, no additional post-processing or iteration is required and the single iteration processing is achieved.

External memory bandwidth is an important issue in system-on-chip (SoC) systems. Especially in high definition (HD) video coding, the bandwidth requirement of off-chip memory is critical in video processing. In recent researches, embedded compression shows high potential on off-chip memory bandwidth reduction. Works about embedded compression have been done for low power applications. However, there is no suitable efficient embedded compression with good rate-distortion performance for high throughput applications. In this paper, an algorithm and hardware architecture of high performance lossy embedded compression is proposed to ease the bus congestion problem while keeping the latency low. Using the proposed algorithm, not only the high throughput requirement of HD video encoder is met, but also the hardware cost is relatively low. From our simulation, about 70% memory bandwidth is reduced with only 0.1 dB PSNR degradation in 1080p HD video.

Multiview video coding (MVC) is getting more and more attention in recent years because of the realistic perception it can bring. However, the current H.264-based coding scheme cannot get enough coding gain in the rate-distortion (R-D) performance. Furthermore, the view synthesis and random access between views are also design challenges. In this paper, a multiview hybrid coding system with texture-depth synthesis is proposed. By sharing the prediction information between texture and depth maps, the depth map can be reconstructed with reduced bitrate overhead. Then, after using matrix-based warping and adaptive disparity compensation, the R-D performance of the proposed system can be about 2 to 4 dB higher than the current MVC coding scheme at low bitrate range.

High Definition (HD) video compression enables vivid reproduction of scenes. However, Motion Estimation (ME) requires large memory capacity and huge memory bandwidth, which are undesirable in many platforms including ASIC and SoC. In this paper, an algorithm and architecture design of cache system and fast ME in HD H.264/AVC are proposed. With the proposed cache system and hardware-oriented fast ME algorithm, the rate-distortion performance is maintained within 0.03dB difference, the size of on-chip memory reduced to only 10% to 21% of original size, while the external memory bandwidth from cache refill is also 18% to 56% less than that of level C data reuse scheme with vertical +64 search range.

This paper presented a novel depth map generation method - the short-term motion assisted color segmentation, which combines the pictorial, monocular and binocular depth cues of human vision. The proposed method utilizes a motion/edge registration technique to avoid the motion jitter error in common motion segmentation. And the motion/image segment adaptation algorithm matches the connected components with the motion segments. Even for static scene, the connected component algorithm is still working for the depth map generation. The experimental results show that the adaptation of motion and image segmentation improves quality and smoothness of the depth map both in the spatial and temporal domain.

Multiview video coding (MVC) plays a critical role in 3D-TV systems. However, MVC require much more memory bandwidth and computational complexity relative to mono-view video coding. Therefore, an efficient prediction scheme is necessary for computation reduction. In this paper, a new fast prediction algorithm, computation-free motion estimation (ME) with inter-view mode decision, is proposed. By utilizing disparity estimation to find corresponding blocks between different views, the coding information, such as modes and motion vectors, can be effectively reused from the coded view channel. Therefore, the computation for ME can be almost totally neglected in most view channels. Besides, the proposed algorithm also provides the computation scalability. Experimental results show that compared with the full search block matching algorithm (FSBMA), the proposed algorithm not only save 61-72% computation with near-FSBMA quality, but save almost 100% computation with a little quality drop of 0.23-0.36 dB. Thus, computation-free ME is achieved.

Multiview video coding (MVC) systems require much more bandwidth and computational complexity relative to mono-view video systems. Thus, when designing a VLSI architecture for MVC systems, the hardware resource allocation is a critical issue. In this paper, we propose a new system bandwidth analysis scheme for various and complicated MVC structures. The precedence constraint in the graph theory is adopted for deriving the processing order of frames in a MVC system. In addition, current block centric scheduling (CBCS) and search window centric scheduling (SWCS) are proposed for MVC bandwidth analysis. By adopting data reuse schemes, several design points are explored with the aid of the proposed analysis scheme. The suitable hardware resource allocation can be easily determined.

Luminance and chrominance correction (LCC) is important in multi-view video coding (MVC) because it provides better rate-distortion performance when encoding video sequences captured by ill-calibrated multi-view cameras. This paper presents a robust and fast LCC algorithm based on motion compensated linear regression which reuses the motion information from the encoder. We adopt the linear weighted prediction model in H.264/AVC as our LCC model. In our experiments, the proposed LCC algorithm outperforms basic histogram matching method up to 0.4dB with only few computational overhead and zero external memory bandwidth. So, the dataflow of this method is suitable for low bandwidth/low power VLSI design for future multi-view applications.

3D video will become noticeable video technology in the next generation. In this paper, a stereo video coding system is proposed from algorithm level to hardware architecture level. We propose a novel stereo video coding system by exploiting joint block compensation scheme to achieve high coding efficiency. It is also suitable for hardware implementation. Due to more than twice computational complexity relative to mono video coding systems, a new hardware architecture based on hierarchical search block matching algorithm (HSBMA) with some modification is proposed. With special data flow, no bubble cycles exist during block matching process. Proposed architecture also adopts near overlapped candidates reuse scheme (NOCRS) to save heavy burden of data access. Besides, by the proposed new scheduling, both on-chip memory requirement and offchip memory bandwidth can be reduced. A prototype chip can achieve real-time requirement under the operating frequency of 81 MHz for 30 D1 frames per second (fps) in left and right channel simultaneously, with ME/DE search range of [-64, +63] in horizontal direction and [-32, +31]/[-16, +15] in vertical direction. Compared with the hardware requirement for implementation of full search block matching algorithm (FSBMA), only 11.5% on-chip SRAM and 1/30 amount of PEs are needed. It shows that the hardware cost is quite small.

The chip design of prediction core in the stereo video hybrid coding system is implemented with 0.18 mum 1P6M technology by TSMC. The die size is 4.53 mm. This IC can achieve real-time requirement under the operating frequency of 81 MHz for 30 D1 frames per second (fps) in the left and the right channel simultaneously, with ME/DE search range of [-64, +63] in horizontal direction and [-32, +31]/[-16, +15] in vertical direction. Compared with the hardware requirement for implementation of full search block matching algorithm (FSBMA), only 11.5% on-chip SRAM and 1/30 amount of PEs are needed. It shows that the hardware cost is quite small

An efficient depth image based rendering with edge dependent depth filter and interpolation is proposed. The proposed method can solve the hole-filling problem in DIBR system efficiently with high quality. The PSNR of the proposed method is better than the previous work by 6 dB and the subjective view shows the quality is better. In addition to that, the number of instruction cycles is 3.7 percent compared with the previous work

Stereo video systems require double bandwidth and more than twice the computational complexity relative to mono-video systems. Thus, an efficient coding scheme is necessary for transmitting stereo video. We propose a novel stereo video coding system by exploiting a joint prediction scheme which combines three prediction schemes to achieve high coding efficiency and low computational complexity. Joint block compensation improves the visual quality. Motion vector prediction and mode pre-decision utilize the features of stereo video to reduce the computational complexity with up to 8-9 times acceleration. Experiments show that the proposed joint prediction scheme is 2 dB better than MPEG-4 TS and 3 dB better than MPEG-4 SP.

An efficient coding system is required for immersive teleconference to transmit stereo video. We propose a novel stereo video coding system by exploiting mesh-based disparity estimation and compensation scheme to achieve high coding efficiency and view synthesis ability. Based on the base-layer-enhance-layer structure, this system can provide stereoscopic scalability and compatibility to standards. With considering asymmetric spatial resolution property, good subjective quality can be achieved in ultra low bitrate situation. A novel fast disparity estimation algorithm named as two-stage iterative block and octagonal matching (TS-IBOM) algorithm is also proposed for this system. Experiments show that the proposed disparity estimation algorithm can generate accurate disparity vectors quickly. It is also shown that the proposed coding system has better coding efficiency than MPEG-4 simple profile and temporal scalability tool. The ultra low bitrate of 69 Kbps can be reached to encode 384/spl times/192 60 fps stereo video.

Disparity estimation is a very important operation in stereo image and video compression. However, existing disparity estimation algorithms require too large computation power. In this paper, we propose a fast disparity estimation algorithm for mesh-based stereo image and video compression with two-stage hybrid approach, which is names as Two Stage Iterative Block and Octagonal Matching algorithm (TS-IBOM). The first stage of this algorithm is Iterative Block Matching algorithm, which can give a good initial guess of the disparity vectors with a little computation. In the second stage, Iterative Octagonal Matching algorithm is employed to refine the disparity vectors. A local updating scheme is also proposed to further accelerate the process by skipping nodes whose disparity vectors have been derived already. Experimental results show that the proposed algorithm can give good results and is 18 times faster than Octagonal Matching. This algorithm is suitable to be employed in mesh-based stereo image and video compression systems.