Search

In order to compress omnidirectional video clips, a projection onto a two-dimensional image plane is necessary. The most commonly used projection format is the equirectangular panoramic projection, which results into a significant amount of redundant samples in the polar areas. The redundant samples incur extra bitrate and increase the encoding/decoding time. In this paper, we study regional down-sampling (RDS) for achieving better compression and smaller encoding/decoding time for omnidirectional content. We extend the persistent RDS method applied equally to all pictures to be applied to selected pictures only in our proposed temporal RDS method and then compare the persistent and temporal RDS methods. The simulation results indicate that both the persistent and temporal RDS improve the rate-distortion (RD) performance compared to the conventional coding of equirectangular panoramas, while the temporal RDS method has less sequence-wise RD performance variation and slightly better RD performance on average when compared to the persistent RDS technique. Alongside the coding methods, we study spherical quality measurement methods for VR images/video and analyze the coding methods with these quality metrics. Moreover, we propose a uniformly sampled spherical quality metric in order to evaluate the coding distortion of omnidirectional videos.

Scalable video coding (SVC) and transmission has been a research topic for many years. Among other objectives, it aims to support different receiving devices, perhaps connected through a heterogeneous network structure, using a single bit stream. Earlier attempts of standardized scalable video coding, for example in MPEG-2, H.263, or MPEG-4 Visual, have not been commercially successful. Nevertheless, the Joint Video Team has recently focused on the development of the scalable video extensions of H.264/AVC, known as SVC. Some of the key problems of older scalable compression techniques have been solved in SVC and, at the same time, new and compelling use cases for SVC have been identified. While it is certainly important to develop coding tools targeted at high coding efficiency, the design of the features of the interface between the core coding technologies and the system and transport are also of vital importance for the success of SVC. Only through this interface, and novel mechanisms defined therein, applications can take advantage of the scalability features of the coded video signal. This paper provides an overview of the system interface features defined in the SVC specification. We discuss, amongst other features, bit stream structure, extended network abstraction layer (NAL) unit header, and supplemental enhancement information (SEI) messages related to scalability information.

A novel method is proposed which minimizes the tune-in time of "channel zapping", i.e. changing from one audiovisual service to another, in IP datacasting (IPDC) over Digital Video Broadcasting for Handheld terminals (DVB-H). DVB-H uses a time-sliced transmission scheme enabling a receiver to turn radio reception off for those time-slices that are not of interest to the user and thus reducing the power consumption used for radio reception. One of the significant factors in tune-in time is the time from the start of media decoding to the start of correct output from decoding, which is minimized when a time-slice starts with a random access point picture such as an independent decoding refresh (IDR) picture in H.264/AVC standard. In IPDC over DVB-H, encapsulation to time-slices is performed independently from encoding in a network element called IP encapsulator. At the time of encoding, time-slice boundaries are typically not known exactly, and it is therefore impossible to govern the location of IDR pictures relative to time-slices. It is proposed that an additional stream consisting of IDR pictures only is transmitted to the IP encapsulator, which replaces pictures in a normal bit stream with IDR pictures according to time-slice boundaries in order to achieve the minimum tune-in delay. Replacing pictures causes a mismatch in the pixel values of the reference pictures between the encoder and decoder and the mismatch error is propagated in the reconstructed video. It has to be ensured that the propagated error is subjectively negligible. Furthermore, the "spliced" stream resulting from the operation of the IP encapsulator should comply with the Hypothetical Reference Decoder (HRD) specification of H.264/AVC. Error propagation caused by the proposed splicing method is analysed and a video rate control system is proposed to satisfy the HRD requirements for the spliced stream. Simulation results show that in addition to fulfilling H.264/AVC compliancy, good average quality of decoded video is achieved with minimum tune-in delay and complexity

Advances in multiview video coding aim to fulfil the required bandwidth and storage capacity of the 3D content. However, despite these evolutions, further compression efficiency can be achieved by taking into account the perceptual characteristics of 3D video. This paper presents a regionally adaptive filtering scheme for 3D video, exploiting the sensitivity of the human visual system to the perceptual characteristics of texture video. The location and the strength of the smoothing filter are determined based on the degree of brightness of each individual pixel within the image. A series of systematic subjective tests were conducted, confirming that no quality degradation is introduced by application of such filtering and hence, keeping the perceived quality of the filtered content intact. At the same time, objective measurements show a Bjontegaard delta bitrate reduction of up to 33.55% and with an average of 21.35%.

The bandwidth and storage restrictions of consumer devices and conventional delivery infrastructures on stereoscopic 3D video require efficient compression methods to save the bandwidth and preserve the perceptive quality at the same time. Compression efficiency is actually determined as the visual quality achieved for a certain amount of bitrate. In this paper, a perception aware coding scheme is proposed to improve the perceived quality of stereoscopic video. The proposed method exploits the sensitivity of human visual system to perceptual characteristics of texture video, to further remove the perceptual redundancy of stereoscopic image pair. The macro blocks within a frame are classified according to their overall brightness, for each of which an adaptive local quantization parameter is selected. The idea is that the darker macro blocks within an image can be compressed with higher ratio without sacrificing the visual quality, especially in the 3D video, where the viewers are more attracted to the pop-out regions and their motion in depth. The proposed scheme has been evaluated by subjective testing, confirming that no quality degradation is introduced by deploying the proposed method. In fact the perceived quality of the stereo content remains intact, while the objective measurements show a Bjontegaard delta bitrate reduction of up to 37.83% and with an average of 27.27%.

Dynamic Adaptive Streaming over HTTP (DASH) has gained wide acceptance due to its ability of bitrate adaptation in diverse network conditions. In the meanwhile, it is asserted that the Scalable Extension (SHVC) of High Efficiency Video Coding (HEVC) can bring more efficient storage and caching compared with traditional single-layer video coding. However, using of SHVC in DASH raises the downstream bitrate relative to encoding each representation independently. This paper proposes a method to improve the downstream rate-distortion performance of SHVC in DASH. Experimental results show that the proposed method can bring up to 16.9 %-unit downstream bitrate reduction compared with directly use SHVC in DASH, while the efficiency in storage space is maintained.

The Dynamic Adaptive Streaming over HTTP (DASH) enables bitrate adaptation through different representations of the same content. It is common to encode random access point (RAP) pictures at segment boundaries to support representation switching. As an open group of pictures (GOP) results into a temporary discontinuity of the video playback due to the inability to decode some pictures when switching representations, closed GOP prediction structures are normally used in DASH. This paper proposes two similar methods for using the open GOP prediction structure in DASH representations while a full picture rate is maintained also during representation switching. The first method is enabled with straightforward changes in the decoding of the High Efficiency Video Coding (H.265/HEVC) standard, whereas the second method utilizes the adaptive resolution change feature of the scalable (SHVC) extension of H.265/HEVC. Experiments show that the proposed methods outperform the use of closed GOPs by 5.6% on average in terms of Bjontegaard delta bitrate (BD-rate).

This paper reviews the multiview extension (MV-HEVC) of the High Efficiency Video Coding (HEVC) standard. MV-HEVC is capable of multiview video coding with or without accompanying depth views. The key design concepts and design elements of MV-HEVC are described in the paper. Furthermore, the features and characteristics of MV-HEVC compared to other standardized video codec extensions for three-dimensional (3D) video coding are reviewed.

The scalable and multiview extensions of the High Efficiency Video Coding share the same high-level syntax coding structure. For the scalable extension, the motion field of the inter-layer reference picture is modified through Motion Field Mapping before used for motion vector prediction. However, the motion field of the inter-layer reference picture is used without modification in the multiview extension. In this paper, a disparity-compensated inter-layer motion prediction is proposed for multiview video coding to achieve disparity compensation in inter-layer motion prediction using the scaled reference layer offset. The experimental results show that comparing with the multiview extension anchor, the proposed method achieves an average of 1.1% bitrate reduction.

The scalable extension of H.264/AVC, known as scalable video coding or SVC, is currently the main focus of the Joint Video Team's work. In its present working draft, the higher level syntax of SVC follows the design principles of H.264/AVC. Self-contained network abstraction layer units (NAL units) form natural entities for packetization. The SVC specification is by no means finalized yet, but nevertheless the work towards an optimized RTP payload format has already started. RFC 3984, the RTP payload specification for H.264/AVC has been taken as a starting point, but it became quickly clear that the scalable features of SVC require adaptation in at least the areas of capability/operation point signaling and documentation of the extended NAL unit header. This paper first gives an overview of the history of scalable video coding, and then reviews the video coding layer (VCL) and NAL of the latest SVC draft specification. Finally, it discusses different aspects of the draft SVC RTP payload format, in- cluding the design criteria, use cases, signaling and payload structure.

The 3rd generation partnership project (3GPP) has defined the protocols and codecs for implementing media streaming services over packet-switched 3G mobile networks. The specification is based on IETF RFCs on audio/video transport. It also adds new features to achieve better adaptation to the mobile network environment. In this paper, we propose an algorithm for handover detection and fast buffer refill that is based on the existing feedback and signaling mechanisms. The proposed algorithm refills the receiver buffer at a faster pace during a limited time frame after a hard handover is detected in order to achieve higher video quality.

H.264/AVC will be an essential component in emerging wireless video applications thanks to its excellent compression efficiency and network-friendly design. However, a video coding standard itself is only one component within the application and transmission environment. Its effectiveness strongly depends on the selection of appropriate modes and parameters at the encoder, at the decoder, as well as in the network. In this paper we introduce the features of the H.264/AVC coding standard that make it suitable for wireless video applications, including features for error resilience, bit rate adaptation, integration into packet networks, interoperability, and buffering considerations. Modern wireless networks provide many different means to adapt quality of service, such as forward error correction methods on different layers and end-to-end or link layer retransmission protocols. The applicability of all these encoding and network features depends on application constraints, such as the maximum tolerable delay, the possibility of online encoding, and the availability of feedback and cross-layer information. We discuss the use of different coding and transport related features for different applications, namely video telephony, video conferencing, video streaming, download-and-play, and video broadcasting. Guidelines for the selection of appropriate video coding tools, video encoder and decoder settings, as well as transport and network parameters are provided and justified. References to relevant research publications and standardization contributions are given.

Different types of prediction are applied in modern video coding. While predictive coding improves compression efficiency, the propagation of transmission errors becomes more likely. In addition, predictive coding brings difficulties to other aspects of video coding, including random access, parallel processing, and scalability. In order to combat the negative effects, video coding schemes introduce mechanisms such as slices and intracoding, to limit and break the prediction. This paper proposes the use of the isolated regions coding tool that jointly limits in-picture prediction and interprediction on a region-of-interest basis. The tool can be used to provide random access points from non-intrapictures and to respond to intrapicture update requests. Furthermore, it can be applied as an error-robust macroblock mode decision method and can be used in combination with unequal error protection. Finally, it enables mixing of scenes, which is useful in coding of masked scene transitions.

The MVC+D standard specifies coding of Multiview Video plus Depth (MVD) data for enabling advanced 3D video applications. MVC+D defines that all views are coded with H.264/MVC encoder at equal spatial resolution. To improve compression efficiency it is possible to use mixed resolution coding in which part of texture views are coded at reduced spatial resolution. In this paper we evaluate the performance of Depth-Based Super Resolution (DBSR) on compressed mixed resolution MVD data. Experimental results show that for sequences with accurate depth data the objective coding performance metric increases. Even though some sequences, with poor depth quality, show slight decrease in coding performance with respect to objective metric, subjective evaluation shows that perceived quality of DBSR method is equal to symmetric resolution case. We also show that depth re-projection consistency check step of the DBSR can be changed to simpler consistency check method. In this way the DBSR computational complexity is reduced by 26 % with 0.2 % dBR average bitrate reduction for coded views and 0.1 % average bitrate increase for synthesized views. We show that proposed scheme outperforms the anchor MVC+D coding scheme by 7.2 % of dBR on average for total coded bitrate and by 10.9 % of dBR on average for synthesized views.

HTTP-based delivery for Video on Demand VoD has been gaining popularity within recent years. With the recently proposed Dynamic Adaptive Streaming over HTTP DASH, video clients may dynamically adapt the requested video quality and bitrate to match their current download rate. To avoid playback interruption, DASH clients attempt to keep the buffer occupancy above a certain minimum level. This mechanism works well for the single view video streaming. For multi-view video streaming application over DASH, the user originates view switching and that only one view of multi-view content is played by a DASH client at a given time. For such applications, it is an open problem how to exploit the buffered video data during the view switching process. In this paper, we propose two fast and efficient view switching approaches in the paradigm of DASH systems, which fully exploit the already buffered video data. The advantages of the proposed approaches are twofold. One is that the view switching delay will be short. The second advantage is that the rate-distortion performance during the view switching period will be high, i.e., using less request data to achieve comparable video playback quality. The experimental results demonstrate the effectiveness of the proposed method.

Viewing while commuting’ is a typical use case for mobilevideo. However, experimental and behavioral influences ofwatching three-dimensional (3D) video in vibratingvehicles have not been widely researched. The goal of thisstudy is 1) to explore the influence of video presentationmodes (two-dimensional and stereoscopic 3D) on thequality of experience and 2) to understand the nature of themovement patterns that users perform to maintain anoptimal viewing position while viewing videos on a mobiledevice in three commuting contexts and in a controlledlaboratory environment. A hybrid method for qualityevaluation was used for combining quantitative preferenceratings, qualitative descriptions of quality, situationalaudio/video data-collection, and sensors. The high-qualityand heterogeneous audiovisual stimuli were viewed on amobile device equipped with a parallax barrier display. Theresults showed that the stereoscopic 3D (S3D) videopresentation mode provided more satisfying quality ofexperience than the two-dimensional presentation mode inall studied contexts. To maintain an optimal viewingposition in the vehicles, the users moved the device in theirhands to the directions around the vertical and thehorizontal axes in a leaned sitting position. This movementbehavior was guided by the contexts but not by the quality,indicating the general importance of these results for mobilevideo viewing in vibrating vehicles.

3D video is composed out of two or more, temporally synchronized, 2D video streams acquired at different camera poses and accompanied by geometrical information. In a mixed resolution 3D video stream, a subset of views is coded at reduced resolution. It has been shown in the literature that subjective quality of mixed resolution 3D video is close to that of full resolution 3D video. In order to improve the coding gain in mixed resolution coding scenario we present a new depth encoding method called view upsampling optimization. A novel depth distortion metric based on the performance of the depth-based super resolution is also presented. Finally, to improve the quality of the decoded video an improved depth-based super resolution method that uses view synthesis quality mapping is used for upsampling of low resolution views. The simulations, performed with the recently standardized MVC+D encoder, show that the proposed solution combined with the state of the art view synthesis distortion outperforms the anchor MVC+D coding scheme by 14.5 % of dBR on average for the total coded bitrate and by 17 % of dBR on average for the synthesized views. acceptedVersion

To meet the requirement of current bandwidth and storage facilities, it is required to further decrease the bitrate of 3D video content. This paper presents a scheme to partially filter different regions of the texture views and depth maps taking into account the characteristics of both of them. A combination of the distance of objects from the cameras and the spatial details of texture within the objects defines the location and strength of the smoothing filter that should be applied to that image. A series of subjective tests was conducted to confirm that the perceived quality of the filtered content remains intact and no quality degradations is introduced by the filtering steps. Finally, objective measurements show a Bjontegaard delta bitrate reduction up to 23.77% with an average of 15.2%.

Asymmetric stereo video coding is a well-known enhancement technique for efficient 3D video rate scaling, taking advantage from the binocular suppression theory. Usually in asymmetric video coding one view is encoded in a higher spatial/ temporal resolution or quality, while the auxiliary view is encoded with a lower resolution or quality. In this paper a novel asymmetric video coding approach is proposed to enhance the stereoscopic video compression efficiency. A regionally adaptive smoothing filter is applied to dark pixels of one view while the same filter is only applied to the light pixels of the other view. The location and the strength of the smoothing filters are determined according to the texture characteristics and the degree of the brightness of each individual pixel within the image. A series of systematic subjective tests were conducted, confirming that no quality degradation is perceptible by application of such filters. This is while the objective measurements show a Bjontegaard delta bitrate reduction of up to 26.6% and with an average of 16.8%.

One approach for stereoscopic video compression is to down sample the content prior to encoding and up sample it to the original spatial resolution after decoding. In this study it is shown that the ratio by which the content should be rescaled is sequence dependent. Hence, a frequency based method is introduced enabling fast and accurate estimation of the best down sampling ratio for different stereoscopic video clips. It is shown that exploiting this approach can bring 3.38% delta bitrate reduction over five camera-captured sequences.

One use case that the scalable extension (SHVC) of the state-of-the-art High Efficiency Video Coding (HEVC) standard aims for is to support Ultra High Definition (UHD) TV broadcast in a backwards compatible way with the existing High Definition (HD) TV broadcast. However, since UHD content typically has higher bit-depth and wider color gamut in addition to increased spatial resolution, the compression efficiency is highly affected by the inter-layer processing applied on the base layer picture. This paper proposes an improvement for the weighted prediction based color gamut scalability to have a better mapping between the color gamuts of the base and enhancement layers. The proposed method aims at capturing the nonlinear characteristics of the color gamut mapping using a piecewise linear model, whose parameters are signaled through weighted prediction mechanism and multiple inter-layer reference pictures. Compared to other existing methods for color gamut mapping in SHVC, such as the 3D Look Up Table (LUT) method, the proposed weighted prediction based approach is less complex, as it does not require any changes to the decoder. The simulation results show up to 3.8% Bjontegaard delta bitrate gain in luma for all intra and 3.0% for random access configurations compared to the existing weighted prediction based scalability method in SHVC.

Traditional forward view synthesis prediction enables the efficient use of depth to provide synthesized frames for texture reference in non-base layers. But asserted drawbacks of high complexity that results from edge detection, hole-filling, up sampling and down sampling in forward warping technique compromise the positive performance. Hence, backward view synthesis prediction is proposed to remove these drawbacks while maintaining the performance. However, fixed depth block used in backward view synthesis prediction limits the performance gain and the number of motion compensation operations, which is a requisite concern of complexity analysis. In this paper, a block based BVSP for inter-layer prediction with only high-level syntax changes is implemented and an adaptive depth block size selection method is proposed. The experimental results show that an average gain of 3.5% bitrate reduction was achieved and after enabling adaptive depth block size selection, this performance gain is relatively maintained while the number of motion compensation operations was reduced to a designated level.

A fundamental challenge of 3D video compression is to provide end users with a high quality video while respecting the limitations of available bandwidth or storage capacities. In this paper, a novel scheme is proposed to filter the texture pictures and depth maps partially targeting a lower bitrate for compression, while preserving the same subjective quality as the one achieved without filtering. In the first step, a depth map is used to calculate a threshold defining the regions of the image where filtering will be applied. This decision is made based on the distance of objects from the camera. The farther the distance of the object, the less likely to have that object be the center of focus for the viewer. In the second step, the content of both texture and depth map belonging to the selected region is filtered. A series of systematic subjective tests showed that the application of such filtering does not decrease the perceived quality of the 3D content compared to the scheme where no filtering is applied. In this case, a BD-rate reduction of up to 17% with an average of 12% is achieved.

Advances in 3D video technology expand the availability of hardware for 3D video generation and display. Some 3D capture and coding arrangements take advantage of a mixed resolution setup. In difference to typical 3D multiview video the mixed resolution scenario assumes that a subset of views is coded at reduced spatial resolution. After decoding, the low resolution views have to upsampled in order to preserve homogeneous resolution of the rendered 3D video. We improve the depth-based super resolution technique that uses a view synthesis process as an essential step. We propose to improve the view synthesis step by per-pixel selection of the projection method. The method is tested on the data coded using a mixed resolution 3D video codec implemented on top of the 3DV-ATM reference software and evaluated under the JCT-3V common test conditions. Simulation results show that our method can improve the depth-based super resolution upsampling by 0.64% of dBR on average for total coded bitrate and 0.55% of dBR on average for synthesized views. Aggregated coding gain with respect to full resolution scenario is improved by 7.71% of dBR on average for total coded bitrate and 11.1% dBR for synthesized views.

Multiview video plus depth (MVD) has proved to be a promising format enabling various 3D applications. One approach to achieve better MVD compression is to adjust the spatial resolution of depth map based on the content and the application. In this research two schemes are considered: first, multiview video coding accompanied by depth maps to improve the texture coding performance. Second, multiview video plus depth (MVD) coding targeting highest quality for synthesized views. Two algorithms to select the best spatial resolution for each scheme are proposed and the results show 10.8% and 16.5% bitrate reduction compared to the anchor case where depth map resolution is fixed.

Color gamut scalability refers to coding a video in a layered manner where the base and enhancement layers are coded in different color gamut spaces. Color gamut scalability and its relationship with spatial scalability are currently being studied for the scalable extension of HEVC (SHVC) to enable coding of ultra-high definition content having BT.2020 color gamut with 10-bit precision as an enhancement layer and high definition content having BT.709 color gamut with 8-bit precision as the base layer. In this paper, we propose to use the weighted prediction tool of the SHVC standard to map the color gamut of the base layer to the enhancement layer. In addition, we also propose a high-precision bit-depth mapping of the base layer to the enhancement layer that jointly performs upsampling with a bit-depth increase. Simulation results show that these two schemes improve the coding efficiency of the All Intra and Random Access configurations by about 6.8% and 3.6% on average, respectively, compared to a basic scheme where the bit-depth of the base layer is increased by simple bit-shifting. These gains are achieved by imposing no changes to the SHVC standard; hence make the proposed method very useful for practical use-cases as well.

The multiview-video-plus-depth (MVD) format selected for 3D video standardization describes 3D scene by video and associated depth and enables generation of virtual views through the depth-image-based rendering (DIBR) process. The Kinect™ sensor is equipped with a red-green-blue (RGB) camera and a depth sensor and it is suitable to capture synchronous video and depth streams. However, these sensors do not produce data with pixel-wise correspondence that is required by the DIBR process.

The multiview-video-plus-depth (MVD) format has become popular in representing 3D video content in a manner that enables free-viewpoint capability in the decoder side. The scalable (SHVC) and multiview (MV-HEVC) extensions of the High Efficiency Video Coding standard (H.265/HEVC) enable their functionality without additional coding tools and use the same high level syntax. In this paper, it is proposed to use the same coding approach as taken in SHVC and MV-HEVC for coding of MVD data. Furthermore, an inter-component motion vector prediction (ICP) method, which is realized through the temporal motion vector prediction (TMVP) mechanism of H.265/HEVC, is introduced to exploit the redundancy between texture and depth views. The experimental results show that 1.0% and 3.6% bitrate reduction can be achieved by ICP compared to independent coding of texture and depth for synthesized views and depth views, respectively, when the ICP method is applied to a 3-view coding scenario intended for multi-view autostereoscopic displays.

This paper presents a novel Multi-View Video plus Depth (MVD) data coding configuration for the 3D video extension of the High Efficiency Video Coding standard (3D-HEVC). To improve compression efficiency of dependent views, 3D-HEVC utilizes disparity information which is derived from the motion information of the neighboring blocks. However, in the case of MVD data coding, disparity information can be derived directly from the depth data. To utilize this approach, we propose to code the depth maps of the dependent views before their associated texture pictures. Consequently, disparity can be derived from coded depth and can be utilized for coding of texture data. The proposed method improves the compression performance of 3D-HEVC by 0.6% in delta bit rate on average. In addition to this, coded depth maps can be used for Backward View Synthesis Prediction (B-VSP) to further reduce bitrate of dependent texture views. Not only the rate distortion performance is improved, but also the proposed configuration reduces the complexity of decoding by over 8%.

Asymmetric stereoscopic video coding, where one view has a higher quality than the other, has been researched widely. In order to balance the quality difference between views, it has been suggested to alternate the higher-quality view periodically. However, it has also been found out that inter-view prediction is more effective when the source of prediction is the higher-quality view. In this paper, it is proposed to change the inter-view prediction direction periodically together with gradual view refresh. The proposal was implemented on top of the reference software for the multiview and multiview-plus-depth extensions of the High Efficiency Video Coding (H.265/HEVC) standard. Experimental results show that up to 9.1% and an average of 3.1% bitrate reduction are achieved with respect to the MPEG Anchor.

In this paper, we propose a practical approach for robust rectification of stereo camera setups without use of calibration pattern. Our solution simplifies the process to a non-general case of rectification to avoid explicit use of Fundamental Matrix estimation. The solution shows better or comparable robustness than some of recent solutions, but for much lower computational cost and code complexity. acceptedVersion

Dynamically changing the spatial resolution in a video conferencing session is useful for seamlessly adapting the bitrate to changing network conditions and for improving the user experience. Similar to earlier standards, the emerging High Efficiency Video Coding (H.265/HEVC) standard does not allow prediction across different resolutions, so an Instantaneous Decoding Refresh (IDR) picture must be sent to reinitialize the stream when a resolution change happens. IDR pictures take significantly more bits compared to predictively coded pictures. Thus, using them for resolution switching significantly reduces coding efficiency and increases the delay. In this paper we propose a method to support efficient adaptive resolution change using the emerging scalable H.265/HEVC standard. The proposed approach utilizes the inter-layer predicted random access pictures at the enhancement layer for resolution switching, instead of IDR pictures. The experimental results show that when the proposed method was used, the bitrate was reduced at the switching point by 34% on average for the tested video sequences. In addition, visual examples are shown demonstrating the improved visual quality with the proposed method.

This work addresses the problem of denoising of range data obtained by ToF continuous signal modulation camera working in low-power mode. The proposed approach is based on non-local mean filtering applied over extensive spatio-temporal block search in complex-valued signal domain. The extensive search allows for using shorter integration times of the range sensors and leads to an effective overcomplete structure suitable for denoising. The filter structure is optimized for real-time operation and achieves O(1) performance for arbitrary patch size by utilizing summed area tables and look-up table data fetching. The experimental results show practically the same performance while compared with state-of-the-art approaches, for greatly improved speed.

This paper describes a novel approach of using depth information for advanced coding of associated video data in Multiview Video plus Depth (MVD)-based 3D video systems. As a possible implementation of this conception, we describe two coding tools that have been developed for H.264/AVC based 3D Video Codec as response to Moving Picture Experts Group (MPEG) Call for Proposals (CfP). These tools are Depth-based Motion Vector Prediction (DMVP) and Backward View Synthesis Prediction (BVSP). Simulation results conducted under JCT-3V/MPEG 3DV Common Test Conditions show, that proposed in this paper tools reduce bit rate of coded video data by 15% of average delta bit rate reduction, which results in 13% of bit rate savings on total for the MVD data over the state-of-the-art MVC+D coding. Moreover, presented in this paper conception of depth-based coding of video has been further developed by MPEG 3DV and JCT-3V and this work resulted in even higher compression efficiency, bringing about 20% of delta bit rate reduction on total for coded MVD data over the reference MVC+D coding. Considering significant gains, proposed in this paper coding approach can be beneficial for development of new 3D video coding standards.

Hypothetical Reference Decoder is a hypothetical decoder model that specifies constraints on the variability of conforming network abstraction layer unit streams or conforming byte streams that an encoding process may produce. High Efficiency Video Coding (HEVC) builds upon and improves the design of the generalized hypothetical reference decoder of H.264/ AVC. This paper describes some of the main improvements of hypothetical reference decoder of HEVC.

Recent developments of three-dimensional (3D) video coding greatly rely on the use of Multiview Video plus Depth (MVD) data format for representing a 3D scene. This type of data can be coded with conventional video compression schemes and can enable advanced 3D video functionality, such as rendering of virtual views at the decoder side. The MVD represents a 3D scene from different viewing angles as video and pixel-wise associated depth data. In this paper we consider the redundancy of depth data in the MVD representation and propose a novel scheme, called Unpaired MVD (UP-MVD) format to be used in 3D video applications. Being a subset of the MVD this new format assumes that a reduced number of depth views compared to the number of texture views can reduce bitrate as well as the encoding/decoding complexity while still providing the 3DV functionality in many scenarios. The simulation results show that the proposed unpaired MVD format outperforms the MVD format on average from 0.9% to 6.95% of Bjontegaard delta bitrate (dBR) for the baseline disparity adjustments from 50% to 100% of the coded baseline, respectively. Moreover, UP-MVD provides equal or better rate-distortion results for all test sequences for up to 20% view separation adjustment, and in five out of seven sequences a better rate-distortion performance is observed when 50% view separation adjustment is applied.

In this paper, we propose a robust and efficient technique for frame-by-frame estimation of disparity ranges in stereo video. The proposed technique utilizes a single-layer image size reduction for faster processing and more effective noise handling. Furthermore, it applies spatial-domain non-linear filtering of both disparity and confidence maps for additional noise suppression and improved range estimation. A mechanism for supporting temporal consistency is proposed as well. Performance comparisons with recent approaches demonstrate the advantages of the proposed approach.

We propose a joint de-noising and data fusion approach where the fused modalities come from conventional high-resolution photo or video camera and low-resolution range sensor of Time-of-flight (ToF) type, operating in restricted conditions of low-emitting power and low number of sensor elements. Our approach includes identifying the various noise sources and suggesting suitable remedies at particular stages of data sensing and fusion. More specifically, fixed pattern noise and system noise are treated at a preliminary denoising stage working on range data only. In contrast to other 2D video/depth fusion approaches, which suggest working in planar coordinates, our approach includes additional denoising refinement in the space of 3D world coordinate system (i.e. point cloud space). Furthermore, the high-resolution grid resampling is performed as an iterative non-uniform to uniform resampling based on the Richardson method. This improves the performance compared to approaches based on low-to-high grid upsampling and subsequent refinement. We report experimental results where the achieved quality of fused data is the same as if the ToF sensor was operating in normal (low-noise) sensing mode.

The emerging MVC+D standard specifies the coding of Multiview Video plus Depth (MVD) data for enabling advanced 3D video applications. MVC+D specifications define the coding of all views of MVD at equal spatial resolution and apply a conventional MVC technique for coding the multiview texture and the depth independently. This paper presents a modified MVC+D coding scheme, where only the base view is coded at the original resolution whereas dependent views are coded at reduced resolution. To enable inter-view prediction, the base view is downsampled within the MVC coding loop to provide a relevant reference for dependent views. At the decoder side, the proposed scheme consists of a post-processing scheme which upsamples of the decoded views to their original resolution. The proposed scheme is compared against the original MVC+D scheme and an average of 4% delta bitrate reduction (dBR) in the coded views and 14.5% of dBR in the synthesized views are reported.

The paper aims at characterizing the role of camera capture process in depth quality estimation by stereo-matching methods. An evaluation software application has been developed which integrates a modular image processing pipeline (IPP) in to the depth estimation process. The application allows for simulating the presence of camera-specific processing artifacts and their influence on the captured stereo imagery in typical imaging scenarios. Furthermore, it allows for benchmarking and optimizing both the depth estimation module and camera capture settings for a jointly-optimal performance.

In asymmetric stereoscopic video compression, the views are coded with different qualities. According to the binocular suppression theory, the perceived quality is closer to that of the higher-fidelity view. Hence, a higher compression ratio is potentially achieved through asymmetric coding. Furthermore, when mixed-resolution coding is applied, the complexity of the coding and decoding is reduced. In this paper, we study whether asymmetric stereoscopic video coding achieves the mentioned claimed benefits. Two sets of systematic subjective quality evaluation experiments are presented in the paper. In the first set of the experiments, we analyze the extent of downsampling for the lower-resolution view in mixed-resolution stereoscopic videos. We show that the lower-resolution view becomes dominant in the subjective quality rating at a certain downsampling ratio, and this is dependent on the sequence, the angular resolution, and the angular width. In the second set of the experiments, we compare symmetric stereoscopic video coding, quality-asymmetric stereoscopic video coding, and mixed-resolution coding subjectively. We show that in many cases, mixed-resolution coding achieves a similar subjective quality to that of symmetric stereoscopic video coding, while the computational complexity is significantly reduced.

We propose a non-local de-noising approach aimed at filtering range data sensed by Photonic Mixer Device sensors. We address specifically the case of poor sensing environment when the reflected signal amplitude is low. In our approach, signal components of phase-delay and amplitude of the sensed signal are regarded as components of a complex-valued variable and processed together in a single step. This imposes better filter adaptivity and similarity weighting. The complex-domain filtering provides additional feedback in the form of improved noise-level confidence, which can be utilized in iterative de-noising schemes. Pre-filtering of individual components is proposed to suppress structural artifacts. Our approach compares favorably with state of the art approaches.

Due to the fact that proxy-driven proxy cache management and the client-driven streaming solution of Dynamic Adaptive Streaming over HTTP (DASH) are two independent processes, some difficulties and challenges arise in media data management at the proxy cache and rate adaptation at the DASH client. This paper presents a novel client-driven joint proxy cache management and DASH rate adaptation method, named CLICRA, which moves prefetching intelligence from the proxy cache to the client. Based on the philosophy of CLICRA, this paper proposes a rate adaptation algorithm, which selects bitrates for the next media segments to be requested by using the predicted buffered media time in the client. CLICRA is realized by conveying information on the segments that are likely to be fetched subsequently to the proxy cache so that it can use the information for prefetching. Simulation results show that the proposed method outperforms the conventional segment-fetch-time-based rate adaptation and the proxy-driven proxy cache management significantly not only in streaming quality at the client but also in bandwidth and storage usage in proxy caches.

Depth maps, an essential part in the new generation of 3D video coding, allow rendering of arbitrary viewpoints of a video scene. Depth maps are characterized by sharp object boundaries, which significantly affect the rendering quality and account for the most bitrate for depth map coding. This paper proposes a novel intra coding method for depth maps based on a two-step adaptive boundary location process. By extracting a series of sub-blocks along a depth boundary and refining the boundary within sub-blocks, accurate predictions for blocks with arbitrary edge shapes can be realized. Experimental results show that the proposed scheme achieves bitrate reductions of up to 28% and 13% on average for seven test sequences of MPEG 3DV compared to original intra coding of H.264/AVC considering the same quality of synthesized views. Besides, subjective quality of virtual views is improved owning to well preserved boundary information.

A Fourier-domain representation of an image exhibits a property where a translation of the image is included in the phase term. This property extends to the Discrete Cosine Transform, where the translation is encoded in the signs of the coefficients. An interpretation of this property for use as a similarity measure in local stereo matching is explored with an efficient way of comparing the transformed blocks to generate a dense disparity map. The method is empirically demonstrated to work also with the Haar wavelet transform, which offers faster computation and improved quality. Results show that presented transform based similarity measures provide better disparity estimates than box aggregated Sum of Absolute Differences and the Census transform when using the percentage of bad pixels as the quality measure. In terms of mean squared error, they achieve better results than SAD but are marginally below Census. The simplicity of making the comparisons results in good scaling with the number of disparity estimates, making the suggested method perform better than SAD also computationally.

With the widespread interest in 3D technology areas such as displays, cameras, and processing, the 3D video is becoming widely available. Due to correlation between views in multiview 3D video at the same temporal location, it is possible to perform video processing operations more efficiently comparing to regular 2D video. In order to improve denoising performance for multiview video, we propose an algorithm based on denoising in 3D DCT domain, which is competitive in performance with state-of-art denoising algorithms and it is suitable for real-time implementation. The proposed algorithm searches for corresponding image patches in temporal and inter-view directions, selects 8 patches with lowest dissimilarity measure, and performs denoising in 3D DCT domain. The novel interview image patch search method brings up to 1.62dB gain in terms of average luma Peak Signal-to-Noise Ratio (PSNR), with average gain 0.6 - 0.8 dB depending on the amount of noise present in test sequences.

Depth-enhanced multiview video, such as the multiview video plus depth (MVD) format, can be used to provide displaying-time view adjustment capability through depth-image-based rendering (DIBR) and additional compression improvement compared to the Multiview Video Coding standard. In this paper a gradual view refresh (GVR) method is presented to code random access points and provide fast startup in streaming for MVD bitstreams. When decoding is started from a GVR point, a subset of the views can be accurately decoded, while the remaining views can be approximately reconstructed using DIBR. Perfect reconstruction of all views can be reached at a subsequent random access point. The GVR coding was found to be effective with up to 10% bitrate reduction for sequences with static camera arrangement. The use of GVR for fast startup in video streaming was found to be clearly superior to transmitting a MVD bitstream conventionally from rate-distortion point of view. It has been found in earlier studies that there seems to be a delay from stimulus onset until depth is fully perceived, hence giving a reason to believe that accurate reconstruction of all views might not be necessary immediately after starting decoding. Furthermore, it was observed in this paper that the objective picture quality reduction during GVR was only moderate, verifying the applicability of the presented GVR method.

There are several data formats available for 3D video, among which is the Multiview Video plus Depth (MVD) representation that enables depth-image-based rendering (DIBR). In addition to DIBR, the MVD data format can enable more efficient compression for texture video coding. In this paper, we propose advanced motion vector prediction (MVP) techniques which utilize the availability of depth information in the MVD format for more efficient coding of the corresponding texture pictures. The proposed MVP was implemented on top of the Multiview Video Coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard and its reference software and tested under the simulation conditions of the Call for Proposals of 3D video coding technology issued by the Moving Picture Experts Group (MPEG). In the provided simulation results the proposed scheme significantly outperformed the conventional MVP (by up to 9.9% and on average by 7.5% in Bjontegaard delta bitrate) when it was applied for coding of 3-view MVD data.

In this study, we applied a locally adaptive filtering in 3D DCT domain to 3 views as a preprocessing stage for encoding. Afterwards, these views were encoded and the decoded views were used by a Depth-Image-Based Rendering algorithm (DIBR) to produce virtual intermediate views. A stereopair at a suitable separation for viewing on a stereoscopic display was selected among the synthesized views. A large-scale subjective assessment of the selected synthesized stereopair was performed. A bitrate reduction of 9% on average and up to 21.8% was achieved with almost no penalties on subjective perceived quality.

We present an approach to measure and model the parameters of human point-of-gaze (PoG) in 3D space. Our model considers the following three parameters: position of the gaze in 3D space, volume encompassed by the gaze and time for the gaze to arrive on the desired target. Extracting the 3D gaze position from binocular gaze data is hindered by three problems. The first problem is the lack of convergence - due to micro saccadic movements the optical lines of both eyes rarely intersect at a point in space. The second problem is resolution - the combination of short observation distance and limited comfort disparity zone typical for a mobile 3D display does not allow the depth of the gaze position to be reliably extracted. The third problem is measurement noise - due to the limited display size, the noise range is close to the range of properly measured data. We have developed a methodology which allows us to suppress most of the measurement noise. This allows us to estimate the typical time which is needed for the point-of-gaze to travel in x, y or z direction. We identify three temporal properties of the binocular PoG. The first is reaction time, which is the minimum time that the vision reacts to a stimulus position change, and is measured as the time between the event and the time the PoG leaves the proximity of the old stimulus position. The second is the travel time of the PoG between the old and new stimulus position. The third is the time-to-arrive, which is the time combining the reaction time, travel time, and the time required for the PoG to settle in the new position. We present the method for filtering the PoG outliers, for deriving the PoG center from binocular eye-tracking data and for calculating the gaze volume as a function of the distance between PoG and the observer. As an outcome from our experiments we present binocular heat maps aggregated over all observers who participated in a viewing test. We also show the mean values for all temporal properties separately for x, y and z direction averaged over all observers. We show the typical size of a binocular area of interest for a portable autostereoscopic display, as well as typical time the 3D vision can react to sudden changes in a 3D scene.