Search

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link. We propose a generative adversarial network for point cloud upsampling, which can not only make the upsampled points evenly distributed on the underlying surface but also efficiently generate clean high frequency regions. The generator of our network includes a dynamic graph hierarchical residual aggregation unit and a hierarchical residual aggregation unit for point feature extraction and upsampling, respectively. The former extracts multiscale point-wise descriptive features, while the latter captures rich feature details with hierarchical residuals. To generate neat edges, our discriminator uses a graph filter to extract and retain high frequency points. The generated high resolution point cloud and corresponding high frequency points help the discriminator learn the global and high frequency properties of the point cloud. We also propose an identity distribution loss function to make sure that the upsampled points remain on the underlying surface of the input low resolution point cloud. To assess the regularity of the upsampled points in high frequency regions, we introduce two evaluation metrics. Objective and subjective results demonstrate that the visual quality of the upsampled point clouds generated by our method is better than that of the state-of-the-art methods.

Marie Skłodowska-Curie Action Project name: Optimized Dynamic Point Cloud Compression, acronym: OPT-PCC, grant number 836192 The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link. In video-based point cloud compression (V-PCC), one geometry video and one color video are generated from a dynamic point cloud. Then, the two videos are compressed independently using a state-of-the-art video coder. In the Moving Picture Experts Group (MPEG) V-PCC test model, the quantization parameters for a given group of frames are constrained according to a fixed offset rule. For example, for the low-delay configuration, the difference between the quantization parameters of the first frame and the quantization parameters of the following frames in the same group is zero by default. We show that the rate-distortion performance of the V-PCC test model can be improved by lifting this constraint and considering the rate-distortion optimization problem as a multi-variable constrained combinatorial optimization problem where the variables are the quantization parameters of all frames. To solve the optimization problem, we use a variant of the differential evolution algorithm. Experimental results for the low-delay configuration show that our method can achieve a Bj{\o}ntegaard delta bitrate of up to -43.04% and more accurate rate control (average bitrate error to the target bitrate of 0.45% vs. 10.75%) compared to the state-of-the-art method, which optimizes the rate-distortion performance subject to the test model default offset rule. We also show that our optimization strategy can be used to improve the rate-distortion performance of two-dimensional video coders.

We propose the first method to predict the Satisfied User Ratio (SUR) for compressed stereo images. The method consists of two main steps. First, considering binocular vision properties, we extract three types of features from stereo images: image quality features, monocular visual features, and binocular visual features. Then, we train a Support Vector Regression (SVR) model to learn a mapping function from the feature space to the SUR values. Experimental results on the SIAT-JSSI dataset show excellent prediction accuracy, with a mean absolute SUR error of only 0.08 for H.265 intra coding and only 0.13 for JPEG2000 compression.

The Publisher's final version can be found by following the DOI link. We modify the video-based point cloud compression standard (V-PCC) by mapping the patches to seven regions and encoding the geometry and color video sequences of each region. We then propose a coarse to fine rate control algorithm for this scheme. The algorithm consists of two major steps. First, we allocate the target bitrate between the geometry and color information. Then, we optimize in turn the geometry and color quantization steps for the video sequences of each region using analytical models for the rate and distortion. Experimental results for eight point clouds showed that the average percent bitrate error of our algorithm is only 3.7%, and its perceptual reconstruction quality is better than that of V-PCC.

This special section contains extended versions of papers presented at the IEEE International Conference on Multimedia & Expo (ICME) in London, July 2020. The invitations to contribute to the special section were based on the quality of the conference paper and its relevance to the aims and scope of the journal. Each submitted manuscript was reviewed by at least three anonymous reviewers. After a rigorous review process, three papers were accepted. Two of them present new developments in immersive media, while the third deals with the problem of efficiently encoding a deep neural network.

International audience; We propose a new approach to text classificationin which we consider the input text as an image and apply2D Convolutional Neural Networks to learn the local andglobal semantics of the sentences from the variations of thevisual patterns of words. Our approach demonstrates thatit is possible to get semantically meaningful features fromimages with text without using optical character recognitionand sequential processing pipelines, techniques that traditionalnatural language processing algorithms require. To validateour approach, we present results for two applications: textclassification and dialog modeling. Using a 2D ConvolutionalNeural Network, we were able to outperform the state-of-art accuracy results for a Chinese text classification task andachieved promising results for seven English text classificationtasks. Furthermore, our approach outperformed the memorynetworks without match types when using out of vocabularyentities from Task 4 of the bAbI dialog dataset.

The Just Noticeable Difference (JND) reveals the minimum distortion that the Human Visual System (HVS) can perceive. Traditional studies on JND mainly focus on background luminance adaptation and contrast masking. However, the HVS does not perceive visual content based on individual pixels or blocks, but on the entire image. In this work, we conduct an interactive subjective visual quality study on the Picture-level JND (PJND) of compressed stereo images. The study, which involves 48 subjects and 10 stereoscopic images compressed with H.265 intra coding and JPEG2000, includes two parts. In the first part, we determine the minimum distortion that the HVS can perceive against a pristine stereo image. In the second part, we explore the minimum distortion that each subject perceives against a distorted stereo image. Modeling the distribution of the PJND samples as Gaussian, we obtain their complementary cumulative distribution functions, which are known as Satisfied User Ratio (SUR) functions. Statistical analysis results demonstrate that the SUR is highly dependent on the image contents. The HVS is more sensitive to distortion in images with more texture details. The compressed stereoscopic images and the PJND samples are collected in a data set called SIAT-JSSI, which we release to the public.

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link. The Satisfied User Ratio (SUR) curve for a lossy image compression scheme, e.g., JPEG, characterizes the probability distribution of the Just Noticeable Difference (JND) level, the smallest distortion level that can be perceived by a subject. We propose the first deep learning approach to predict such SUR curves. Instead of the direct approach of regressing the SUR curve itself for a given reference image, our model is trained on pairs of images, original and compressed. Relying on a Siamese Convolutional Neural Network (CNN), feature pooling, a fully connected regression-head, and transfer learning, we achieved a good prediction performance. Experiments on the MCL-JCI dataset showed a mean Bhattacharyya distance between the predicted and the original JND distributions of only 0.072.

Rate-distortion optimal 3D point cloud compression is very challenging due to the irregular structure of 3D point clouds. For a popular 3D point cloud codec that uses octrees for geometry compression and JPEG for color compression, we first find analytical models that describe the relationship between the encoding parameters and the bitrate and distortion, respectively. We then use our models to formulate the rate-distortion optimization problem as a constrained convex optimization problem and apply an interior point method to solve it. Experimental results for six 3D point clouds show that our technique gives similar results to exhaustive search at only about 1.57% of its computational cost.

We propose a framework for energy-based human activity recognition in a household environment. We apply machine learning techniques to infer the state of household appliances from their energy consumption data and use rule- based scenarios that exploit these states to detect human activity. Our decision engine achieved a 99.1% accuracy for real-world data collected in the kitchens of two smart homes.

Multiview plus depth (MVD) is an emerging video format with many applications, including 3-D television and free viewpoint television. During the broadcast of a compressed MVD video, transmission errors may cause the loss of whole frames, resulting in significant degradation of video quality. Error concealment techniques have been widely used to deal with transmission errors in video communication. However, the existing solutions do not address the requirement that the reconstructed frames should be consistent with neighboring frames, i.e., corresponding pixels should have consistent color information. We propose a new consistency model for error concealment of MVD video that allows one to maintain a high level of consistency between frames of the same view (temporal consistency) and those of neighboring views (inter-view consistency). We then propose an algorithm that uses our model to implement concealment in a consistent way. Simulations with the reference software for the multiview video coding project of the joint video team of the ISO/IEC MPEG and ITU-T VCEG show that our method outperforms benchmark techniques, including a baseline approach based on the boundary matching algorithm, with respect to both reconstruction quality and view consistency.

View synthesis prediction (VSP) is a coding mode that predicts video blocks from synthesised frames. It is particularly useful in a multi-camera setup with large inter-camera distances. Adding a VSP-based SKIP mode to a standard Multiview Video Coding (MVC) framework improves the rate-distortion (RD) performance but increases the time complexity of the encoder. This letter proposes an early mode decision technique for VSP SKIP-enhanced MVC. Our method uses the correlation between the RD costs of the VSP SKIP mode in neighbouring views and Bayesian decision theory to reduce the number of candidate coding modes for a given macroblock. Simulation results showed that our technique can save up to 36.20% of the encoding time without any significant loss in RD performance.

International audience; We propose a peer-to-peer system for streaming user-generated live video. Peers are arranged in levels so that video is delivered at about the same time to all peers in the same level, and peers in a higher level watch the video before those in a lower level. We encode the video bitstream with rateless codes and use trees to transmit the encoded symbols. Trees are constructed to minimize the transmission rate for the source while maximizing the number of served peers and guaranteeing on-time delivery and reliability at the peers. We formulate this objective as a height bounded spanning forest problem with nodal capacity constraint and compute a solution using a heuristic polynomial-time algorithm. We conduct ns-2 simulations to study the trade-off between used bandwidth and video quality for various packet loss rates and link latencies.

©20009 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder. We consider a state of the art system that uses layered source coding and forward error correction with Reed- Solomon codes to efficiently transmit 3D meshes over lossy packet networks. Given a transmission bit budget, the performance of this system can be optimized by determining how many layers should be sent, how each layer should be packetized, and how many parity bits should be allocated to each layer such that the expected distortion at the receiver is minimum. The previous solution for this optimization problem uses exhaustive search, which is not feasible when the transmission bit budget is large.We propose instead an exact algorithm that solves this optimization problem in linear time and space. We illustrate the advantages of our approach by providing experimental results for the CPM (Compressed Progressive Meshes) mesh compression technique DFG Research Training Group GK-1042.

When streaming packetized media data over a lossy packet network, it is desirable to use transmission strategies that minimize the expected distortion subject to a constraint on the expected transmission rate. Because the computation of such optimal strategies is usually an intractable problem, fast heuristic techniques are often used. We first show that when the graph that gives the decoding dependencies between the data packets is reducible to a tree, optimal transmission strategies can be efficiently computed with dynamic programming algorithms. The proposed algorithms are much faster than other exact algorithms developed for arbitrary dependency graphs. They are slower than previous heuristic techniques but can provide much better solutions. We also show how to apply our algorithms to find high-quality approximate solutions when the dependency graph is not tree reducible. To validate our approach, we run simulations for MPEG1 and H.264 video data. We first consider a simulated packet erasure channel. Then we implement a real video streaming system and provide experimental results for an Internet connection.

Given a number of available layers of source data and a transmission bit budget, we propose an algorithm that determines how many layers should be sent and how many protection bits should be allocated to each transmitted layer such that the expected distortion at the receiver is minimum. The algorithm is used for robust transmission of progressively compressed 3D models over a packet erasure channel. In contrast to the previous approach, which uses exhaustive search, the time complexity of our algorithm is linear in the transmission bit budget.

A list Viterbi algorithm (LVA) finds the n most likely paths in a trellis diagram of a convolutional code. One of the most efficient LVAs is the tree-trellis algorithm of Soong and Huang. We propose a new implementation of this algorithm. Instead of storing the candidate paths in a single list sorted according to the metrics of the paths, we show that it is computationally more efficient to use several unsorted lists, where all paths of the same list have the same metric. For an arbitrary integer bit metric, both the time and space complexity of our implementation are linear in n. Experimental results for a binary symmetric channel and an additive white Gaussian noise channel show that our implementation is much faster than all previous LVAs.

We consider the problem of ratedistortion optimized streaming of packetized multimedia data over a single quality of service network using feedback and retransmissions. For a single data unit, we prove that the problem is NP-hardand provide efficient branch and bound algorithms that are much faster than the previously best solution based on dynamic programming. For a group of interdependent data units, we show how to compute optimal solutions with branch and bound algorithms. The branch and bound algorithms for a group of data units are much slower than the current state of the art, a heuristic technique known as sensitivity adaptation. However, in many realworld situations, they provide a significantly better rate-distortion performance.

Reliable real-time transmission of packetized embedded multimedia data over noisy channels requires the design of fast error control algorithms. For packet erasure channels, efficient forward error correction is obtained by using systematic Reed-Solomon (RS) codes across packets. For fading channels, state-of-the-art performance is given by a product channel code where each column code is an RS code and each row code is a concatenation of an outer cyclic redundancy check code and an inner rate-compatible punctured convolutional code. For each of these two systems, we propose a low-memory linear-time iterative improvement algorithm to compute an error protection solution. Experimental results for the two-dimensional and three-dimensional set partitioning in hierarchical trees coders showed that our algorithms provide close to optimal average peak signal-to-noise ratio performance, and that their running time is significantly lower than that of all previously proposed solutions.

We propose a system for packet loss protection of quality scalable video coders. The compressed bitstream is composed of a non-scalable base layer and an embedded enhancement layer as in MPEG4-fine granularity scalability (FGS) and the emerging H26L-FGS. But the system can also be used with embedded video bitstreams like the 3D-SPIHT bitstream. Our system allocates a given transmission bitrate between the base layer bitstream and the enhancement layer bitstream in an efficient way using feedback. The base layer is protected with equal packet loss protection. For the enhancement layer, unequal packet loss protection is used to provide graceful degradation of video quality in the presence of packet erasure. Preliminary results for a hypothetical operational distortion-rate function and the 3D-SP1HT video coder show promising performance of the proposed system.

We consider the problem of rate-distortion optimal streaming of packetized media with sender-driven transmission over a single-QoS network using feedback and retransmissions. For a single data unit, we prove that the problem is NP-hard and provide efficient branch and bound algorithms that are in practice much faster than the best known solution. For a group of interdependent data units, we show how to compute optimal solutions with branch and bound algorithms. The branch and bound algorithms for a group of data units are slower than the current state of the art, the heuristic sensitivity adaptation algorithm, but provide a significantly better rate-distortion performance in many real-world situations.

A novel hybrid scheme combining fractal image compression with mean-removed shape-gain vector quantization is presented. The scheme uses a small set of VQ codebook blocks as a block classifier for the domain blocks and as an alternative means of coding when able to provide a satisfying distortion. Our scheme is shown to improve the performance of conventional fractal coding in all its aspects. The rate-distortion curve is ameliorated, and both the encoding and the decoding are faster.

A novel hybrid scheme combining fractal image compression with mean-removed shape-gain vector quantization is presented. The algorithm, based on a distance classification fractal coder with fixed cluster centers, decides whether to encode a range block by a cluster center or by a domain block. Our scheme is shown to improve the performance of conventional fractal coding in all its aspects. The rate-distortion curve is ameliorated, and both the encoding and the decoding are faster.

Optimal fractal coding consists of finding, in a finite set of contractive affine mappings, one whose unique fixed point is closest to the original image. Optimal fractal coding is an NP-hard combinatorial optimization problem. Conventional coding is based on a greedy suboptimal algorithm known as collage coding. In a previous study, we proposed a local search algorithm that significantly improves on collage coding. However the algorithm, which requires the computation of many fixed points, is computationally expensive. In this paper we provide techniques that drastically reduce the time complexity of the algorithm.

Summary form only given. We present a new AVQ-based video coder for very low bitrates. To encode a block from a frame, the encoder offers three modes: (1) a block from the same position in the last frame can be taken; (2) the block can be represented with a vector from the codebook; or (3) a new vector, that sufficiently represents a block, can be inserted into the codebook. For mode 2 a mean-removed VQ scheme is used. The decision on how blocks are encoded and how the codebook is updated is done in an rate-distortion (R-D) optimized fashion. The codebook of shape blocks is updated once per frame. First results for an implementation of such a scheme have been reported previously. Here we extend the method to incorporate a wavelet image transform before coding in order to enhance the compression performance. In addition the rate-distortion optimization is comprehensively discussed. Our R-D optimization is based on an efficient convex-hull computation. This method is compared to common R-D optimizations that use a Lagrangian multiplier approach. In the discussion of our R-D method we show the similarities and differences between our scheme and the generalized threshold replenishment (GTR) method of Fowler et al. (1997). Furthermore, we demonstrate that the translation of our R-D optimized AVQ into the wavelet domain leads to an improved coding performance. We present coding results that show that one can achieve the same encoding quality as with comparable standard transform coding (H.263). In addition we offer an empirical analysis of the short- and long-term behavior of the adaptive codebook. This analysis indicates that the AVQ method uses the vectors in its codebook for some kind of long-term prediction.