Search

In this Special Section of the IEEE Transactions on Circuits and Systems for Video Technology, it is our honor to introduce the Versatile Video Coding (VVC) standard, the latest of the historic partnership collaborations between the International Telecommunication Union Telecommunication Standardization Sector (ITU-T), the International Organization for Standardization (ISO), and the International Electrotechnical Commission (IEC) in the field of video coding standardization.

Aim To develop a consensus paper on the central points of an international invitational think-tank on nursing and artificial intelligence (AI). Methods We established the Nursing and Artificial Intelligence Leadership (NAIL) Collaborative, comprising interdisciplinary experts in AI development, biomedical ethics, AI in primary care, AI legal aspects, philosophy of AI in health, nursing practice, implementation science, leaders in health informatics practice and international health informatics groups, a representative of patients and the public, and the Chair of the ITU/WHO Focus Group on Artificial Intelligence for Health. The NAIL Collaborative convened at a 3-day invitational think tank in autumn 2019. Activities included a pre-event survey, expert presentations and working sessions to identify priority areas for action, opportunities and recommendations to address these. In this paper, we summarize the key discussion points and notes from the aforementioned activities. Implications for nursing Nursing's limited current engagement with discourses on AI and health posts a risk that the profession is not part of the conversations that have potentially significant impacts on nursing practice. Conclusion There are numerous gaps and a timely need for the nursing profession to be among the leaders and drivers of conversations around AI in health systems. Impact We outline crucial gaps where focused effort is required for nursing to take a leadership role in shaping AI use in health systems. Three priorities were identified that need to be addressed in the near future: (a) Nurses must understand the relationship between the data they collect and AI technologies they use; (b) Nurses need to be meaningfully involved in all stages of AI: from development to implementation; and (c) There is a substantial untapped and an unexplored potential for nursing to contribute to the development of AI technologies for global health and humanitarian efforts.

Machine learning (ML) and artificial intelligence (AI) methods hold great potential for healthcare, for example, for purposes of diagnosis or prognosis that include a wide range of pattern recognition tasks. Ensuring that health ML/AI models are trustworthy will consequently become increasingly important soon. The ITU/WHO focus group on "AI for Health" is working on validation standards for health AI that can help to assess the quality of the powerful but complex technologies in a comparable and transparent manner. In particular, standardized benchmarking can serve as a valuable tool to determine the merits and limits of different health ML/AI models. In this article, ongoing work of the ITU/WHO initiative is introduced and set into perspective with related digital health and AI standardization efforts.

The performance of visual quality prediction models is commonly assumed to be closely tied to their ability to capture perceptually relevant image aspects. Models are thus either based on sophisticated feature extractors carefully designed from extensive domain knowledge or optimized through feature learning. In contrast to this, we find feature extractors constructed from random noise to be sufficient to learn a linear regression model whose quality predictions reach high correlations with human visual quality ratings, on par with a model with learned features. We analyze this curious result and show that besides the quality of feature extractors also their quantity plays a crucial role - with top performances only being achieved in highly overparameterized models., To be published at ICIP 2021

The PSNR and MSE are the computationally simplest and thus most widely used measures for image quality, although they correlate only poorly with perceived visual quality. More accurate quality models that rely on processing on both the reference and distorted image are potentially difficult to integrate in time-critical communication systems where computational complexity is disadvantageous. This paper derives the concept of distortion sensitivity as a property of the reference image that compensates for a given computational quality model a potential lack of perceptual relevance. This compensation method is applied to the PSNR and leads to a local weighting scheme for the MSE. Local weights are estimated by a deep convolutional neural network and used to improve the PSNR in a computationally graceful distribution of computationally complex processing to the reference image only. The performance of the proposed estimation approach is evaluated on LIVE, TID2013 and CSIQ databases and shows comparable or superior performance compared to benchmark image quality measures.

Objectives The number of studies employing artificial intelligence (AI), specifically machine and deep learning, is growing fast. The majority of studies suffer from limitations in planning, conduct and reporting, resulting in low robustness, reproducibility and applicability. We here present a consented checklist on planning, conducting and reporting of AI studies for authors, reviewers and readers in dental research. Methods Lending from existing reviews, standards and other guidance documents, an initial draft of the checklist and an explanatory document were derived and discussed among the members of IADR’s e-oral network and the ITU/WHO focus group “Artificial Intelligence for Health (AI4H)”. The checklist was consented by 27 group members via an e-Delphi process. Results Thirty-one items on planning, conducting and reporting of AI studies were agreed on. These involve items on the studies’ wider goal, focus, design and specific aims, data sampling and reporting, sample estimation, reference test construction, model parameters, training and evaluation, uncertainty and explainability, performance metrics and data partitions. Conclusion Authors, reviewers and readers should consider this checklist when planning, conducting, reporting and evaluating studies on AI in dentistry. Clinical significance Current studies on AI in dentistry show considerable weaknesses, hampering their replication and application. This checklist may help to overcome this issue and advance AI research as well as facilitate a debate on standards in this fields.

This Conference Presentation, "Lossless coding support in VVC" was recorded for the Optics + Photonics Digital Forum.

The Intra Subpartition (ISP) mode is one of the intra prediction tools incorporated to the new Versatile Video Coding (VVC) standard. ISP divides a luma intra-predicted block along one dimension into 2 or 4 smaller blocks, called subpartitions, that are predicted using the same intra mode. This paper describes the design of this tool and its encoder search implementation in the VVC Test Model 7.3 (VTM-7.3) software. The main challenge of the ISP encoder search is the fact that the mode pre-selection based on the sum of absolute transformed differences typically utilized for intra prediction tools is not feasible in the ISP case, given that it would require knowing beforehand the values of the reconstructed samples of the subpartitions. For this reason, VTM employs a different strategy aimed to overcome this issue. The experimental tool-off tests carried out for the All Intra configuration show a gain of 0.52% for the 22-37 Quantization Parameter (QP) range with an associated encoder runtime of 85%. The results are improved to a 1.06% gain and an 87% encoder runtime in the case of the 32-47 QP range. Analogously, for the tool-on case the results for the 22-37 QP range are a 1.17% gain and a 134% encoder runtime and this improves in the 32-47 QP range to a 1.56% gain and a 126% encoder runtime.

Federated Learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit it’s popularity, it has been observed that Federated Learning yields suboptimal results if the local clients’ data distributions diverge. The recently proposed Clustered Federated Learning Framework addresses this issue, by separating the client population into different groups based on the pairwise cosine similarities between their parameter updates. In this work we investigate the application of CFL to byzantine settings, where a subset of clients behaves unpredictably or tries to disturb the joint training effort in an directed or undirected way. We perform experiments with deep neural networks on common Federated Learning datasets which demonstrate that CFL (without modifications) is able to reliably detect byzantine clients and remove them from training.

With the growing demand for deploying Deep Learning models to the “edge”, it is paramount to develop techniques that allow to execute state-of-the-art models within very tight and limited resource constraints. In this work we propose a software-hardware optimization paradigm for obtaining a highly efficient execution engine of deep neural networks (DNNs) that are based on fully-connected layers. The work’s approach is centred around compression as a means for reducing the area as well as power requirements of, concretely, multilayer perceptrons (MLPs) with high predictive performances. Firstly, we design a novel hardware architecture named FantastIC4, which (1) supports the efficient on-chip execution of multiple compact representations of fully-connected layers and (2) minimizes the required number of multipliers for inference down to only 4 (thus the name). Moreover, in order to make the models amenable for efficient execution on FantastIC4, we introduce a novel entropy-constrained training method that renders them to be robust to 4bit quantization and highly compressible in size simultaneously. The experimental results show that we can achieve throughputs of 2.45 TOPS with a total power consumption of 3.6W on a Virtual Ultrascale FPGA XCVU440 device implementation, and achieve a total power efficiency of 20.17 TOPS/W on a 22nm process ASIC version. When compared to other state-of-the-art accelerators designed for the Google Speech Command (GSC) dataset, FantastIC4 is better by $51\times $ in terms of throughput and $145\times $ in terms of area efficiency (GOPS/mm2).

Healthcare can benefit considerably from advanced information processing technologies, in particular from machine learning (ML) and artificial intelligence (AI). However, the health domain only hesitantly adopts these powerful but complex innovations so far, because any technical fault can affect people's health, privacy, and consequently their entire lives. In this paper, we substantiate that international standards are required for thoroughly validating AI solutions for health, by benchmarking their performance. These standards might ultimately create well-founded trust in those AI solutions that have provided conclusive evidence to be accurate, effective and reliable. We give reasons that standardized benchmarking of AI solutions for health is a necessary complement of established assessment procedures. In particular, we demonstrate that it is beneficial to tackle this topic on a global scale and summarize the achievements of the first year of the ITU/WHO focus group on “AI for Health” that has tasked itself to work towards creating these evaluation standards.

In most practical situations, the compression or transmission of images and videos creates distortions that will eventually be perceived by a human observer. Vice versa, image and video restoration techniques, such as inpainting or denoising, aim to enhance the quality of experience of human viewers. Correctly assessing the similarity between an image and an undistorted reference image as subjectively experienced by a human viewer can thus lead to significant improvements in any transmission, compression, or restoration system. This paper introduces the Haar wavelet-based perceptual similarity index (HaarPSI), a novel and computationally inexpensive similarity measure for full reference image quality assessment. The HaarPSI utilizes the coefficients obtained from a Haar wavelet decomposition to assess local similarities between two images, as well as the relative importance of image areas. The consistency of the HaarPSI with the human quality of experience was validated on four large benchmark databases containing thousands of differently distorted images. On these databases, the HaarPSI achieves higher correlations with human opinion scores than state-of-the-art full reference similarity measures like the structural similarity index (SSIM), the feature similarity index (FSIM), and the visual saliency-based index (VSI). Along with the simple computational structure and the short execution time, these experimental results suggest a high applicability of the HaarPSI in real world tasks.

Accurate estimation of visual quality as perceived by humans is crucial for modern multimedia systems and, given the evident ease for humans, a surprisingly difficult task for computers. Complexity considerations as imperative for real-time applications render this problem even more challenging. This paper studies the application of a neural network-based spatial model of distortion sensitivity to the quality prediction of spatio-temporal videos. We propose a simple yet effective adaptation of the loss function to cope with saturation effects in human quality ratings. This adaptation drastically decreases the number of iterations necessary for training networks to replicate psychophysical human responses. Our experimental results show significantly improved prediction performance of the spatio-temporal PSNR when compensated for spatial distortion sensitivity while maintaining the advantage of low complexity.

The development of the emerging Versatile Video Coding (VVC) standard was motivated by the need of significant bit-rate reductions for natural video content as well as content for different applications, such as computer generated screen content. The signal characteristics of screen content video are different to the ones of natural content. These include sharp edges as well as at areas of the same color. In block-based hybrid video coding designs, as employed in VVC and its predecessors standards, skipping the transform stage of the prediction residual for screen content signals can be beneficial due to the different residual signal characteristics. In this paper, a modified transform coefficient level coding tailored for transform skip residual signals is presented. This includes no signaling of the last significant position, a coded block ag for every subblock, modified context modeling and binarization as well as a limit for the number of context coded bins per sample. Experimental results show bit-rate savings up to 3.45% and 9.55% for two different classes of screen content test sequences coded in a random access configuration.

The peak signal-to-noise ratio (PSNR) is the most used objective measure for assessing perceptual image quality when it comes to image and video compression tasks, despite the fact that it exhibits weak performance in reflecting human perception. To address this problem, many image quality assessment (IQA) methods were proposed, e. g. the structural similarity quality measure (SSIM) and its extension, the multi-scale SSIM (MS-SSIM). In this paper we revisit and evaluate a block-based perceptually weighted PSNR (WPSNR) which calculates weighting factors to capture visual sensitivity of local image regions. We further introduce a sample-based version of WPSNR which determines those sensitivity weights with higher spatial accuracy. These methods are computationally inexpensive compared to other similarity measures and are shown to outperform PSNR, SSIM and similar perceptual quality measures when it comes to approximate subjective ratings of JPEG or JPEG2000 compressed images.

It is well known that input-invariant quantization in perceptual image or video coding often leads to visually suboptimal results and that quantization parameter adaptation (QPA) based on a model of the human visual system can improve subjective coding quality. This paper introduces a simple low-complexity QPA algorithm, controlled using a block-wise perceptually weighted distortion measure representing a generalization of the PSNR metric. The weighting scheme of this WPSNR metric is based on a psychovisual model. It directly leads to a perceptually adapted scaling of the block-wise Lagrange parameter used in the bit-allocation process in the encoder and, consequently, to a block-wise QPA. Unlike prior QPA approaches, the proposal avoids classifications of picture regions and easily extends from still-image or grayscale to video or chromatic coding. The WPSNR metric also uses fewer algorithmic operations than e. g. the multiscale structural similarity measure (MS-SSIM). Due to the results of two formal subjective tests indicating its visual benefit, the QPA proposal has been adopted into VTM, the currently developed Versatile Video Coding (VVC) reference software.

One of the innovations in H.265/HEVC is the quad-tree partitioning framework. It allows flexible block subdivision and mode allocation across the encoded picture. The increased flexibility comes at a cost of vast search space expansion, making exhaustive search algorithms inapplicable. We propose a novel early termination condition to skip the exhaustive search of whole tree-branches in the well-established top-down encoding approach. The condition is based on a simple and intuitive split cost prediction. It can be parametrized to control the trade-off between the speed-up and caused BD-rate loss. Data driven parameter estimation and parameter number reduction is presented. For random-access encoding, the method can achieve an average speed-up of 30% with a BD-rate loss of 0.03%. At another trade-off point, speed-up is increased to over 40% for a BD-rate loss below 0.5%.

In a template matching (TM) intra method, the neighboring samples of the current block are regarded as a template. The decoder searches for the best template match in the current reconstructed picture using an error minimizing metric like sum of squared differences (SSD). The prediction signal is generated by copying the samples of the adjacent block to the selected template match. The increased decoder complexity from the search algorithm makes it less attractive for modern applications like video calling. In a previous publication [1], we presented a region-based template matching (RTM) approach for intra coding. Compared to the conventional TM methods which searches for the template match in a complete search window, RTM searches in a region of the search window. Thus, RTM offers a better trade-off between coding efficiency and decoder complexity. Nevertheless, the memory requirements and number of computations to be carried at the decoder are still high, making RTM difficult for hardware realization. This paper aims to address these issues.

More than three quarters of all bits transmitted today over the consumer Internet are video data. Accordingly, video data is of major importance for the digital transformation. The related field of digital video processing has played a key role in establishing successful products and services in a wide range of sectors including communications, health, industry, autonomous vehicles and security technology. Particularly in the entertainment sector, video data has shaped the mass market via services like HD and UHD TV or streaming services. For these applications, efficient transmission only became feasible through video coding with methods of efficient compression. In addition, production systems and processing techniques for highly realistic dynamic 3D video scenes have been developed. In these key areas, the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institute, HHI is playing a worldwide leading role, in particular in the key areas of video coding and 3D video processing, also through successfully contributing to video coding standardization.

Mobile communications have permanently changed our society and our ways of communicating ever since the global availability of mobile speech services, and because these communications form the basis for mobile Internet. They have facilitated a new dimension of productivity growth and manufacturing and service process networking since the use of the Internet. Their technical basis is founded on a deep understanding of the relationships between radio and telecommunications technology, beginning with radio wave propagation and modeling, through techniques for digital signal processing and a scalable system design for a cellular radio system with mobility support, to methods for system analysis and optimization. The Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institute, HHI has been working in the field of mobile telephony communications for 20 years and has made key contributions to the third, fourth, and fifth generations. Alongside research articles and numerous first-time demonstrations of key technological components, the institute is also an active contributor to 3GPP standardization.

For glasses-free 3-D viewing, autostereoscopic displays provide a dense set of views showing a scene from slightly different viewpoints. As the number of views and their spatial positions are display dependent, a generic depth-enhanced multi-view video plus depth format is used for coding and transmission. From this, the required views are generated via depth image-based rendering (DIBR) techniques by using original views and associated depth information. A critical problem in the DIBR techniques is that texture regions, which are covered by foreground objects in the original image, may become visible in the synthesized view, especially in extrapolation scenarios. To cover these disoccluded areas, a novel hybrid texture synthesis method is described in this paper. Our approach combines a new patch-based and a novel fast parametric texture synthesis method in an innovative way to compute the disoccluded image areas. The proposed method outperforms state-of-the-art approaches in terms of peak signal-to-noise ratio, structural similarity, visual saliency induced index, feature similarity index (color), and run-time.

This paper presents an advanced depth intra-coding approach for 3D video coding based on the High Efficiency Video Coding (HEVC) standard and the multiview video plus depth (MVD) representation. This paper is motivated by the fact that depth signals have specific characteristics that differ from those of natural signals, i.e., camera-view video. Our approach replaces conventional intra-picture coding for the depth component, targeting a consistent and efficient support of 3D video applications that utilize depth maps or polygon meshes or both, with a high depth coding efficiency in terms of minimal artifacts in rendered views and meshes with a minimal number of triangles for a given bit rate. For this purpose, we introduce intra-picture prediction modes based on geometric primitives along with a residual coding method in the spatial domain, substituting conventional intra-prediction modes and transform coding, respectively. The results show that our solution achieves the same quality of rendered or synthesized views with about the same bit rate as MVD coding with the 3D video extension of HEVC (3D-HEVC) for high-quality depth maps and with about 8% less overall bit rate as with 3D-HEVC without related depth tools. At the same time, the combination of 3D video with 3D computer graphics content is substantially simplified, as the geometry-based depth intra signals can be represented as a surface mesh with about 85% less triangles, generated directly in the decoding process as an alternative decoder output.

Due to its computational simplicity, the PSNR is a popular and widely used image quality measure, although it correlates poorly with perceived visual quality. Distortion sensitivity, a reference image specific property, can be used to compensate for the lack of perceptual relevance of the PSNR. Based on the functional mapping between perceptual and computational quality a deep convolutional neural network is used to estimate patchwise distortion sensitivity. The local estimates are used for an imagewise perceptual adaptation of the PSNR. The performance of the proposed estimation approach is evaluated on the LIVE and TID2013 databases and shows comparable or superior performance as compared to benchmark image quality measures.

Today’s hybrid video coding systems typically perform an intra-picture prediction whereby blocks of samples are predicted from previously decoded samples of the same picture. For example, HEVC uses a set of angular prediction patterns to exploit directional sample correlations. In this paper, we propose new intra-picture prediction modes whose construction consists of two steps: First, a set of features is extracted from the decoded samples. Second, these features are used to select a predefined image pattern as the prediction signal. Since several intra prediction modes are proposed for each block-shape, a specific signalization scheme is also proposed. Our intra prediction modes lead to significant coding gains over state of the art video coding technologies.

Modern perceptual image coders reach impressively high subjective quality even at low bit-rates but tend to denoise or “detexturize” the coded pictures. Traditionally, two independent parametric approaches, known as texture and film grain synthesis, have been applied in the spatial domain as pre and post-processors around the codec to counteract such effects. In this work, a unified alternative, operating directly within the spectral domain of conventional transform codecs with tight coupling to the transform coefficient quantizer, is proposed. Due to its design, this spectrally adaptive noise filling tool (SANFT) enables highly input adaptive realizations by reusing the coder's existing optimized spatial and spectral partitioning algorithms. Formal subjective evaluation in the context of a main still picture” High Emciency Video Coding (HEVC) implementation confirms the benefit of the proposal.

In video coding, in-loop filtering has attracted attention due to its increasing coding performances. In this paper the shearlet-based loop filter is proposed using a sparsifying transform, the shearlet transform, which can identify the important structures of natural images such as edges in the sparse transform domain. This allows for separating efficiently the important information from noise components. Our novel approach for in-loop filtering is to apply a shearlet transform to the decoded image, separating important structures from noise and perform an inverse shearlet transform combined with Wiener filtering. This effectively removes compression artefacts due to quantization noise and keeps the important features of the original image. Simulation results show that our shearlet based loop filter can improve the state-of-the-art video coding standard HEVC through up to 10.5% bit rate reduction along with improved subjective visual quality.

In video coding, adaptive loop filter (ALF) has attracted attention due to its increasing coding performances. Recently ALF has been further developed for its extension, which introduces geometry transformation-based adaptive loop filter (GALF) outperforming the existing ALF techniques. The main idea of ALF is to apply a classification to obtain multiple classes, which gives a partition of a set of all pixel locations. After that, a Wiener filter is applied for each class. Therefore, the performance of ALF essentially relies on how its classification behaves. In this paper, we introduce a novel classification method, Multiple feature-based Classifications ALF (MCALF) extending a classification in GALF and show that it increases coding efficiency while only marginally raising encoding complexity. The key idea is to apply more than one classifier at the encoder to group all reconstructed samples and then to select a classifier with the best RD-performance to carry out the classification process. Simulation results show that around 2% bit rate reduction can be achieved on top of GALF for some selected test sequences.

Steady-state visual evoked potentials (SSVEP) are brain responses elicited by periodic visual stimuli. Recently it was shown that the use of SSVEP in quality studies allows for accurate psychophysiological assessment of perceived visual quality, but the influence of the stimulation frequency is still unclear. This paper studies experimentally the relation between the SNR of the neural signal and the stimulation frequency in an psychophysiological quality assessment setup. For various source images tested at different distortion magnitudes over the range of 6 different stimulation frequencies, we show physiologically plausible results that provide insights into the temporal dynamics of neural distortion processing. Our findings inform a rational choice of stimulation frequency in SSVEP-based image quality assessment studies. This potentially improves the experimental setup of future image quality assessment studies exploiting the SSVEP paradigm.

Steady-state visual evoked potentials (SSVEPs) are neural responses, measurable using electroencephalography (EEG), that are directly linked to sensory processing of visual stimuli. In this paper, SSVEP is used to assess the perceived quality of texture images. The EEG-based assessment method is compared with conventional methods, and recorded EEG data are correlated to obtained mean opinion scores (MOSs). A dimensionality reduction technique for EEG data called spatio-spectral decomposition (SSD) is adapted for the SSVEP framework and used to extract physiologically meaningful and plausible neural components from the EEG recordings. It is shown that the use of SSD not only increases the correlation between neural features and MOS to $r=-0.93$ , but also solves the problem of channel selection in an EEG-based image-quality assessment.

In an objective approach for the assessment of quality of experience the neural correlates of EEG data are studied when stereoscopic images are presented in three different conditions containing vertical disparity. These conditions are compared to a similar image in 2D both on the channel level by studying the ERP components and on the source level by the localization of the corresponding ERP component. Our findings posit that P1 component in the occipital cortex has significantly increased in amplitude for 3D condition without vertical disparity compared to the 2D condition. According to previous studies, this component increases when depth information are added to the stimulus which is in line with our findings. However the amplitude of this component has significantly decreased for 3D condition with maximum vertical disparity compared to the 3D condition without vertical disparity. We have concluded that the perception of stereoscopic depth by subjects have decreased in this case due to the distortion introduced by vertical disparity. The underlying sources corresponding to P1 component are localized. Except for the power differences, the source locations do not differ for different conditions.

Although being one of the simplest and most widely used image quality metrics (IQMs) the peak signal-to-noise ratio (PSNR) correlates only poorly with visual quality as perceived by humans. Based on an analysis of the non-linear mapping from PSNR to mean opinion scores (MOS) we identify a functional mapping parameter to adapt the PSNR perceptually meaningful. Neurophysiologically motivated, a shearlet-based correction is proposed for controlling this perceptual PSNR adaption. The performance of the proposed perceptually adapted PSNR is evaluated on the LIVE and TID2013 databases and shows to be superior or comparable to benchmark IQMs.

In todays video compression systems, the encoder typically follows an optimization procedure to find a compressed representation of the video signal. While primary optimization criteria are bit rate and image distortion, low complexity of this procedure may also be of importance in some applications, making complexity a third objective. We approach this problem by treating the encoding procedure as a decision process in time and make it amenable to reinforcement learning. Our learning algorithm computes a strategy in a compact functional representation, which is then employed in the video encoder to control its search. By including measured execution time into the reinforcement signal with a lagrangian weight, we realize a trade-off between RD-performance and computational complexity controlled by a single parameter. Using the reference software test model (HM) of the HEVC video coding standard, we show that over half the encoding time can be saved at the same RD-performance.

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach pro vides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

The compression capability of several generations of video coding standards is compared by means of peak signal-to-noise ratio (PSNR) and subjective testing results. A unified approach is applied to the analysis of designs, including H.262/MPEG-2 Video, H.263, MPEG-4 Visual, H.264/MPEG-4 Advanced Video Coding (AVC), and High Efficiency Video Coding (HEVC). The results of subjective tests for WVGA and HD sequences indicate that HEVC encoders can achieve equivalent subjective reproduction quality as encoders that conform to H.264/MPEG-4 AVC when using approximately 50% less bit rate on average. The HEVC design is shown to be especially effective for low bit rates, high-resolution video content, and low-delay communication applications. The measured subjective improvement somewhat exceeds the improvement measured by the PSNR metric.

This paper presents a no reference image (NR) quality assessment (IQA) method based on a deep convolutional neural network (CNN). The CNN takes unpreprocessed image patches as an input and estimates the quality without employing any domain knowledge. By that, features and natural scene statistics are learnt purely data driven and combined with pooling and regression in one framework. We evaluate the network on the LIVE database and achieve a linear Pearson correlation superior to state-of-the-art NR IQA methods. We also apply the network to the image forensics task of decoder-sided quantization parameter estimation and also here achieve correlations of r = 0.989.

This paper proposes a reduced reference image quality assessment method using only a low number of features. It involves a shearlet decomposition, directional pooling of the obtained coefficient and extracts the scalewise statistical location parameter as a feature. The proposed method is tested and compared to similar approaches on the LIVE image database. On this database it outperforms the compared methods on five of seven distortion types and on the full testset with a linear correlation of = 0.89.

Three experiments addressing the assessment of perceived image quality in a patch-based manner are compared for HEVC compression artifacts. It is shown that image patches of a size small as 128×128 pixel are large enough to evaluate the perceived image quality in a Degradation Category Rating (DCR) setting. Ratings obtained with 128×128 pixel sized images patches and 512×512 pixel sized images of the same spatial statistics show a correlation of r=0.99. Based on this finding, image quality assessment of 128×128 pixel sized image patches degraded by HEVC compression is compared for controlled lab environment and uncontrolled crowdsourcing settings. Although we find high overall correlation between the quality ratings obtained in the two environments, observers tend to give worse ratings in the crowdsourcing setting and for conditions of higher quality a reduction of correlation is observed. These findings have implications for choosing controlled vs. uncontrolled viewing conditions for image quality assessment for real-life applications.

This paper presents a full-reference (FR) image quality assessment (IQA) method based on a deep convolutional neural network (CNN). The CNN extracts features from distorted and reference image patches and estimates the perceived quality of the distorted ones by combining and regressing the feature vectors using two fully connected layers. The CNN consists of 12 convolution and max-pooling layers; activation is done by a rectifier activation function (ReLU). The overall IQA score is computed by aggregating the patch quality estimates. Three different feature combination methods and two aggregation approaches are proposed and evaluated in this paper. Experiments are performed on the LIVE and TID2013 databases. On both databases linear Pearson correlations superior to state-of-the-art IQA methods are achieved.

We present a deep neural network-based approach to image quality assessment (IQA). The network is trained end-to-end and comprises ten convolutional layers and five pooling layers for feature extraction, and two fully connected layers for regression, which makes it significantly deeper than related IQA models. Unique features of the proposed architecture are that: 1) with slight adaptations it can be used in a no-reference (NR) as well as in a full-reference (FR) IQA setting and 2) it allows for joint learning of local quality and local weights, i.e., relative importance of local quality to the global quality estimate, in an unified framework. Our approach is purely data-driven and does not rely on hand-crafted features or other types of prior domain knowledge about the human visual system or image statistics. We evaluate the proposed approach on the LIVE, CISQ, and TID2013 databases as well as the LIVE In the wild image quality challenge database and show superior performance to state-of-the-art NR and FR IQA methods. Finally, cross-database evaluation shows a high ability to generalize between different databases, indicating a high robustness of the learned features.

This paper discusses about video telephony. So equipment manufacturers turned to the corporate world, introducing pricey videoconferencing systems designed to replace on-site meetings and reduce travel costs. While many companies invested in the technology in the 1990s, it typically gathered dust, unused. It looked as though people didn't want it anywhere.

A unified optimization framework for rate allocation among multiple video multicast sessions sharing a wireless network is presented. Our framework applies to delivery of both scalable and non-scalable video streams. In both cases, the optimization objective is to minimize the total video distortion of all peers without incurring excessive network utilization. Our system model explicitly accounts for heterogeneity in wireless link capacities, traffic contention among neighboring links, as well as different video rate-distortion (RD) characteristics. The proposed distributed rate allocation scheme leverages cross-layer information exchange between the media access control and application layers to achieve fast convergence at the optimal, media-aware allocation. Performance of the proposed media-aware rate allocation protocol is compared against a heuristic scheme based on TCP-friendly rate control (TFRC). In network simulations of standard-definition video streaming over single or multiple multicast trees, the proposed scheme consistently achieves higher overall video quality than the TFRC-based heuristics. When delivering scalable streams, the flexibility of per-peer rate adaptation inside each multicast tree yields a further slight improvement in overall video quality over multicast of non-scalable streams.

A depth image-based rendering (DIBR) approach with advanced inpainting methods is presented. The DIBR algorithm can be used in 3-D video applications to synthesize a number of different perspectives of the same scene, e.g., from a multiview-video-plus-depth (MVD) representation. This MVD format consists of video and depth sequences for a limited number of original camera views of the same natural scene. Here, DIBR methods allow the computation of additional new views. An inherent problem of the view synthesis concept is the fact that image information which is occluded in the original views may become visible, especially in extrapolated views beyond the viewing range of the original cameras. The presented algorithm synthesizes these occluded textures. The synthesizer achieves visually satisfying results by taking spatial and temporal consistency measures into account. Detailed experiments show significant objective and subjective gains of the proposed method in comparis on to the state-of-the-art methods.

Media delivery, especially video delivery over mobile channels may be affected by transmission bitrate variations or temporary link interruptions caused by changes in the channel conditions or the wireless interface. In this paper, we present the use of Prioritybased Media Delivery (PMD) for Scalable Video Coding (SVC) to overcome link interruptions and channel bitrate reductions in mobile networks by performing a transmission scheduling algorithm that prioritizes media data according to its importance. The proposed approach comprises a priority-based media pre-buffer to overcome periods under reduced connectivity. The PMD algorithm aims to use the same transmission bitrate and overall buffer size as the traditional streaming approach, yet is more likely to overcome interruptions and reduced bitrate periods. PMD achieves longer continuous playback than the traditional approach, avoiding disruptions in the video playout and therefore improving the video playback quality. We analyze the use of SVC with PMD in the traditional RTP streaming and in the adaptive HTTP streaming context. We show benefits of using SVC in terms of received quality during interruption and re-buffering time, i.e. the time required to fill a desired pre-buffer at the receiver. We present a quality optimization approach for PMD and show results for different interruption/ bitrate-reduction scenarios.

The ITU-T and ISO/IEC standard for scalable video coding was recently finalized. SVC allows for scalability of the video bitstream in the temporal, spatial, or fidelity domain, or any combination of those. Video scalability may be used for different purposes, such as saving bandwidth when the same media content is required to be sent simultaneously on a broadcast medium at different resolutions to support heterogeneous devices, when unequal error protection shall be used for coverage extension in wireless broadcasting, as well as for rate shaping in IPTV environments. Furthermore, it may also be useful in layered multicast transmission over the Internet or peer-to-peer networks, or in any transmission scenario where prioritized transmission for network flows is meaningful. In order to make usage of SVC in the aforementioned use cases, standards for defining the transport format and procedure are required. Therefore, we give a detailed overview of the recently finished SVC standards on transport over IP/RTP and the MPEG-2 transport stream. Both standards are important for IPTV and video on demand, where the first is important for SVC transport over mobile broadcast/multicast channels, and the latter is also important for SVC transport over traditional digital broadcast channels.

Scenarios for the use of the recently approved Scalable Video Coding (SVC) extension of H.264/MPEG4-AVC in IPTV services are presented. For that, a brief technical overview of SVC when deployed in IPTV services is provided. The coding efficiency of the various scalability types of SVC is demonstrated followed by an analysis of the complexity of the various SVC tools. Based on this technical characterization, it is described how the different SVC features such as efficient methods for graceful degradation, bit rate adaptation, and format adaptation, can be mapped to application requirements of IPTV services. It is discussed how such mappings can lead to improved content portability, management and distribution as well as an improved management of access network throughput resulting in better quality of service and experience for the users of IPTV services.

This is the author's accepted manuscript. The final published article is available from the link below. Copyright @ IEEE 2008. This paper proposes a novel method for the assessment of picture quality, called triple stimulus continuous evaluation scale (TSCES), to allow the direct comparison of different HDTV formats. The method uses an upper picture quality anchor and a lower picture quality anchor with defined impairments. The HDTV format under test is evaluated in a subjective comparison with the upper and lower anchors. The method utilizes three displays in a particular vertical arrangement. In an initial series of tests with the novel method, the HDTV formats 1080p/50,1080i/25, and 720p/50 were compared at various bit-rates and with seven different content types on three identical 1920 times 1080 pixel displays. It was found that the new method provided stable and consistent results. The method was tested with 1080p/50,1080i/25, and 720p/50 HDTV images that had been coded with H.264/AVC High profile. The result of the assessment was that the progressive HDTV formats found higher appreciation by the assessors than the interlaced HDTV format. A system chain proposal is given for future media production and delivery to take advantage of this outcome. Recommendations for future research conclude the paper.

An experimental analysis of multiview video coding (MVC) for various temporal and inter-view prediction structures is presented. The compression method is based on the multiple reference picture technique in the H.264/AVC video coding standard. The idea is to exploit the statistical dependencies from both temporal and inter-view reference pictures for motion-compensated prediction. The effectiveness of this approach is demonstrated by an experimental analysis of temporal versus inter-view prediction in terms of the Lagrange cost function. The results show that prediction with temporal reference pictures is highly efficient, but for 20% of a picture's blocks on average prediction with reference pictures; from adjacent views is more efficient. Hierarchical B pictures are used as basic structure for temporal prediction. Their advantages are combined with inter-view prediction for different temporal hierarchy levels, starting from simulcast coding with no inter-view prediction up to full level inter-view prediction. When using inter-view prediction at key picture temporal levels, average gains of 1.4-dB peak signal-to-noise ratio (PSNR) are reported, while additionally using inter-view prediction at nonkey picture temporal levels, average gains of 1.6-dB PSNR are reported. For some cases, gains of more than 3 dB, corresponding to bit-rate savings of up to 50%, are obtained.