Search

This paper presents the overview and rationale behind the Decoder-Side Depth Estimation (DSDE) mode of the MPEG Immersive Video (MIV) standard, using the Geometry Absent profile, for efficient compression of immersive multiview video. A MIV bitstream generated by an encoder operating in the DSDE mode does not include depth maps. It only contains the information required to reconstruct them in the client or in the cloud: decoded views and metadata. The paper explains the technical details and techniques supported by this novel MIV DSDE mode. The description additionally includes the specification on Geometry Assistance Supplemental Enhancement Information which helps to reduce the complexity of depth estimation, when performed in the cloud or at the decoder side. The depth estimation in MIV is a non-normative part of the decoding process, therefore, any method can be used to compute the depth maps. This paper lists a set of requirements for depth estimation, induced by the specific characteristics of the DSDE. The depth estimation reference software, continuously and collaboratively developed with MIV to meet these requirements, is presented in this paper. Several original experimental results are presented. The efficiency of the DSDE is compared to two MIV profiles. The combined non-transmission of depth maps and efficient coding of textures enabled by the DSDE leads to efficient compression and rendering quality improvement compared to the usual encoder-side depth estimation. Moreover, results of the first evaluation of state-of-the-art multiview depth estimators in the DSDE context, including machine learning techniques, are presented. [ABSTRACT FROM AUTHOR]

In this paper, a data assimilation network is proposed to tackle the challenges of domain generalization for person re-identification (ReID). Most of the existing research efforts only focus on single-dataset issues, and the trained models are difficult to generalize to unseen scenarios. This paper presents a distinctive idea to improve the generality of the model by assimilating three types of images: style-variant images, misaligned images and unlabeled images. The latter two are often ignored in the previous domain generalization ReID studies. In this paper, a non-local convolutional block attention module is designed for assimilating the misaligned images, and an attention adversary network is introduced to correct it. A progressive augmented memory is designed for assimilating the unlabeled images by progressive learning. Moreover, we propose an attention adversary difference loss for attention correction, and a labeling-guide discriminative embedding loss for progressive learning. Rather than designing a specific feature extractor that is robust to style shift as in most previous domain generalization work, we propose a data assimilation meta-learning procedure to train the proposed network, so that it learns to assimilate style-variant images. It is worth mentioning that we add an unlabeled augmented dataset to the source domain to tackle the domain generalization ReID tasks. Extensive experiments demonstrate that our approach significantly outperforms the state-of-the-art domain generalization methods. [ABSTRACT FROM AUTHOR]

Image classification needs to consider the existence of image degradations in practice. Although degraded images have various levels of degradation, the degradation levels are usually unknown. This paper proposes a convolutional neural network to classify degraded images by using a restoration network and an ensemble learning. The proposed network can automatically infer ensemble weights by using estimated degradation levels of degraded images and features of restored images, where the degradation levels are estimated internally. The proposed network is mainly discussed with JPEG distortion, while degradations of both Gaussian noise and blurring are also examined. We demonstrate that the proposed network can classify degraded images over various levels of degradation. This paper also reveals how the image-quality of training data for a classification network affects the classification performance of degraded images. [ABSTRACT FROM AUTHOR]

As one of the fundamental technologies for scene understanding, semantic segmentation has been widely explored in the last few years. Light field cameras encode the geometric information by simultaneously recording the spatial information and angular information of light rays, which provides us with a new way to solve this issue. In this paper, we propose a high-quality and challenging urban scene dataset, containing 1074 samples composed of real-world and synthetic light field images as well as pixel-wise annotations for 14 semantic classes. To the best of our knowledge, it is the largest and the most diverse light field dataset for semantic segmentation. We further design two new semantic segmentation baselines tailored for light field and compare them with state-of-the-art RGB, video and RGB-D-based methods using the proposed dataset. The outperforming results of our baselines demonstrate the advantages of the geometric information in light field for this task. We also provide evaluations of super-resolution and depth estimation methods, showing that the proposed dataset presents new challenges and supports detailed comparisons among different methods. We expect this work inspires new research direction and stimulates scientific progress in related fields. The complete dataset is available at https://github.com/HAWKEYE-Group/UrbanLF. [ABSTRACT FROM AUTHOR]

Translation averaging is known to be more difficult than rotation averaging due to scale ambiguity, estimation sensitivity, and solution uncertainty. Existing approaches have exposed their limitations in terms of accuracy, robustness, simplicity, or efficiency. To tackle this tough problem, a simple yet effective translation averaging pipeline, termed as Incremental Translation Averaging (ITA), is proposed in this paper. It combines the advantages of high accuracy and robustness in incremental parameter estimation pipeline and the advantages of high simplicity and efficiency in global motion averaging approach. Unlike the traditional translation averaging methods which estimate all the absolute camera locations simultaneously and suffer from inaccuracy in parameter estimation and incompleteness in scene reconstruction, our ITA computes them novelly in an incremental way with higher accuracy and robustness. Thanks to the introduction of incremental parameter estimation thought into the translation averaging pipeline, 1) our ITA is robust to measurement outliers and accurate in parameter estimation; and 2) our ITA is simple and efficient because of its less dependency on complicated optimization, carefully-designed preprocessing, or additional information. Comprehensive evaluations on the 1DSfM dataset demonstrate the effectiveness of our ITA and its advantages over several state-of-the-art translation averaging approaches. [ABSTRACT FROM AUTHOR]

Deep learning has greatly promoted the development of multi-view stereo in recent years. However, how to measure the reliability of the estimated depth map for practical applications and make reasonable depth hypothesis sampling for the cost volume building in the coarse-to-fine architecture are still unresolved crucial problems. To this end, an Uncertainty Guided multi-view Network (UGNet) is proposed in this paper. In order to enable the network to perceive the uncertainty, an uncertainty-aware loss function is introduced, which not only can infer uncertainty implicitly in an unsupervised manner but also can reduce the bad impact of high uncertainty regions and the erroneous labels in the training set during training. Moreover, an uncertainty-based depth hypothesis sampling strategy is further proposed to adaptively determine the depth search range of each pixel for finer stages, which helps to generate more rational depth intervals compared with other methods and build more compact cost volumes without redundancy. Experimental results on DTU dataset, BlendedMVS dataset, Tanks and Temples dataset and ETH3D high-res benchmark show that our method achieves promising reconstruction results compared with other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Recently, deep semi-supervised hashing methods have attracted increasing attention, which can significantly improve retrieval performance by leveraging abundant unlabeled data. These methods usually generate surrogate supervision signals to learn with unlabeled data, such as neighborhood information and augmentation invariant requirements. However, an essential issue of these methods is that the supervised signals are not always reliable, which may damage the performance. In this paper, we propose a novel Uncertainty-Aware and Multi-Granularity Consistent Constrained Semi-Supervised Hashing (UMCSH) method to alleviate the negative effects of noisy supervised signals and enlarge the inter-class distance. Specifically, our UMCSH mainly consists of an Uncertainty-Aware Instance-Level Consistency (UAILC) model and a Cluster-Based Class-Level Consistency (CBCLC) model. UAILC introduces an uncertainty estimation method to select reliable supervised signals to extract discriminative features for each unlabeled data. CBCLC establishes connections between labeled data and unlabeled data by encouraging each unlabeled sample to be close to the hash center (calculated with the labeled data) according to its pseudo-label. Extensive experimental results demonstrate the superior performance of our proposed approach compared with several state-of-the-art semi-supervised hashing methods. [ABSTRACT FROM AUTHOR]

Cross-modal hashing that leverages hash functions to project high-dimensional data from different modalities into the compact common hamming space, has shown immeasurable potential in cross-modal retrieval. To ease labor costs, unsupervised cross-modal hashing methods are proposed. However, existing unsupervised methods still suffer from two factors in the optimization of hash functions: 1) similarity guidance, they barely give a clear definition of whether is similar or not between data points, leading to the residual of the redundant information; 2) optimization strategy, they ignore the fact that the similarity learning abilities of different hash functions are different, which makes the hash function of one modality weaker than the hash function of the other modality. To alleviate such limitations, this paper proposes an unsupervised cross-modal hashing method to train hash functions with discriminative similarity guidance and adaptively-enhanced optimization strategy, termed Deep Adaptively-Enhanced Hashing (DAEH). Specifically, to estimate the similarity relations with discriminability, Information Mixed Similarity Estimation (IMSE) is designed by integrating information from distance distributions and the similarity ratio. Moreover, Adaptive Teacher Guided Enhancement (ATGE) optimization strategy is also designed, which employs information theory to discover the weaker hash function and utilizes an extra teacher network to enhance it. Extensive experiments on three benchmark datasets demonstrate the superiority of the proposed DAEH against the state-of-the-arts. [ABSTRACT FROM AUTHOR]

As a specific data hiding technique, reversible data hiding (RDH) has recently received extensive attention. By this technique, both the embedded data and the original cover image can be exactly extracted from the marked image. In our previous work, a general expansion-shifting model for RDH is proposed by introducing the so-called reversible embedding function (REF). With REF, RDH can be designed and the corresponding rate-distortion formulations can be established, providing an approach to optimize the reversible embedding performance. In this paper, by extending our previous work, optimal REF for one-dimensional histogram is investigated, and all optimal REF are derived in this case when the maximum modification to the cover pixel is limited as a small value. Moreover, based on the derived optimal REF for one-dimensional histogram and multiple histograms modification, a practical RDH scheme is presented and it is experimental verified better than some state-of-the-art algorithms in terms of capacity-distortion performance. [ABSTRACT FROM AUTHOR]

With the widespread use of deep learning in single object tracking task, mainstream tracking algorithms treat tracking as a combined classification and regression problem. Classification aims at locating an arbitrary target, and regression aims at estimating the corresponding bounding box. In this paper, we focus on regression and propose a novel box estimation network, which consists of a transformer encoder target pyramid guide (TPG) and transformer decoder target pyramid spread (TPS). Specifically, the transformer encoder TPG is designed to generate fine-grained prior knowledge with explicit representation for template targets. In contrast to the raw transformer encoder, we capture the visual dependence through local-global self-attention and deem the multi-scale target regions as the “local” region. Using this fine-grained prior knowledge, we design the transformer decoder TPS to spread it to the subsequent search regions with high affinity to accurately estimate the bounding boxes. Considering that self-attention fails to model information interaction across channels between the template target and search regions, we develop a channel-wise cross-attention block within the TPS as compensation. Extensive experiments on the OTB100, UAV123, NFS, VOT2020, VOT2021, LaSOT, LaSOT_ext, TrackingNet and GOT-10k benchmarks show that the proposed box estimation network outperforms most existing box estimation methods. Furthermore, our trackers based on this estimation network exhibit a competitive performance against state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

Event cameras such as the Dynamic Vision Sensor (DVS) are useful because of their low latency, sparse output, and high dynamic range. In this paper, we propose a DVS+FPGA camera platform and use it to demonstrate the hardware implementation of event-based corner keypoint detection and adaptive block-matching optical flow. To adapt sample rate dynamically, events are accumulated in event slices using the area event count slice exposure method. The area event count is feedback controlled by the average optical flow matching distance. Corners are detected by streaks of accumulated events on event slice rings of radius 3 and 4 pixels. Corner detection takes about 6 clock cycles (16 MHz event rate at the 100MHz clock frequency) At the corners, flow vectors are computed in 100 clock cycles (1 MHz event rate). The multiscale block match size is $25\times 25$ pixels and the flow vectors span up to 30-pixel match distance. The FPGA processes the sum-of-absolute distance block matching at 123 GOp/s, the equivalent of 1230 Op/clock cycle. EDFLOW is several times more accurate on MVSEC drone and driving optical flow benchmarking sequences than the previous best DVS FPGA optical flow implementation, and achieves similar accuracy to the CNN-based EV-Flownet, although it burns about 100 times less power. The EDFLOW design and benchmarking videos are available at https://sites.google.com/view/edflow21/home. [ABSTRACT FROM AUTHOR]

Discriminative correlation filter (DCF)-based methods have demonstrated superior performance in UAV tracking via fusing multiple types of features and updating models online. However, most DCF-based trackers simply cascade different features, failing to fully take advantage of their complementary strength. In addition, online update strategies are limited to using a single and fixed learning rate, which often leads to model degradation when suffering tracking challenges. In this paper, we present an Auto-Perceiving Correlation Filter (APCF) which explicitly models the target and context with a novel Target State and Background Perception (TSBP) feature. Concretely, we first propose a simple yet effective State Evaluation Metric (SEM) to estimate target states by analyzing the spatial distribution of responses. Based on SEM, we extract TSBP features by adaptively selecting effective features depending on the current target state. Accordingly, a new online model update strategy is also introduced to avoid model degradation. Moreover, we further introduce a perception regularization term to make the extracted feature emphasis more on the target rather than background. Extensive experiments on four widely-used UAV benchmarks have well demonstrated the superiority of the proposed method compared with both DCF and deep learning based trackers while running at a high speed of 76.7 FPS on a single CPU. In addition, APCF with deep features also performs favorably against state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

Continuous and accurate depth information of blast furnace burden surface is important for optimizing charging operations, thereby reducing its energy consumption and CO2 emissions. However, depth estimation for a single image is challenging, especially when estimating the depth of burden surface images in the harsh internal environment of the blast furnace. In this paper, a novel method that is based on edge defocus tracking is proposed to estimate the depth of burden surface images with different morphological characteristics. First, an endoscopic video acquisition system is designed, key frames of burden surface video in stable state are extracted based on feature point optical flow method, and the sparse depth is estimated by using the defocus-based method. Next, the burden surface image is divided into four subregions according to the distribution characteristics of the burden surface, the edge line trajectories and an eight-direction depth gradient template are designed to develop depth propagation rules. Finally, the depth is propagated from edge to the entire image based on edge line tracking method. The experimental results show that the proposed method can accurately and efficiently estimate the depth of the burden surface and provide key data support for optimizing the operation of blast furnace. [ABSTRACT FROM AUTHOR]

Given a face image, most of previous works in gaze estimation infer the gaze via a well-trained model with supervised training. However, the distribution of test data may be very different compared to that of training data since samples might be corrupted in real-world scenarios (e.g., taking a photo in strong light). This will lead to a gap between source domain (i.e., training data) and target domain (i.e., test data). In this paper, we first introduce self-supervised learning into our method for addressing challenging situations in gaze estimation. Moreover, existing appearance-based gaze estimation methods focus on directing towards the development of powerful regressors, which mainly utilize face and eye images simultaneously or face (eye) images only. However, the problem of inter cues between face and eye features has been largely overlooked. To this end, we propose a novel Modulation-based Adaptive Network (MANet) for gaze estimation, which uses high-level knowledge to filter the distractive information and bridges the intrinsic relationship between face and eye features. Further, we combine self-supervised learning and MANet to learn to adapt to challenging cases, such as abnormal lighting conditions and poor-quality images, by minimizing a self-supervised loss and a supervised loss jointly. The experimental results on several datasets demonstrate the effectiveness of our proposed approach with a real-time speed of 900 fps on a PC with an NVIDIA Titan RTX GPU. [ABSTRACT FROM AUTHOR]

Monocular depth estimation, as one of the fundamental tasks of computer vision, plays a crucial role in three-dimensional (3D) scene understanding and perception. Usually, deep learning methods recover monocular depth maps using continuous regression manners by minimizing the errors between the ground-truth depth and the predicted depth. However, fine depth features may not be fully captured through layer-by-layer coding, which is prone to low spatial resolution depth maps and insufficient details. Furthermore, it usually converges slowly and suffers from unsatisfactory results. To tackle these issues, we propose a novel model, named context-based ordinal regression network (CORNet), to reconstruct monocular depth maps in the ordinal regression manner with context information in this paper. Firstly, we put forward a novel context-based encoder with a feature transformation (FT) module to learn context information and details from inputs, and output multi-scale feature maps. Then, we design a boundary enhancement module (BEM) with a spatial attention mechanism following each operation of feature fusion, which captures boundary features in the scene to enhance the border depth. Finally, a feature optimization module (FOM) is designed to fuse and optimize the multi-scale features and boundary features to strengthen depth learning. What’s more, we introduce an ordinal weighted inference to predict depth maps from probabilities and discretization values. Experiments and results on two challenging datasets, KITTI and NYU Depth V2, demonstrate that our proposed CORNet can estimate monocular depth maps effectively and obtain superior performance in capturing geometric features over existing methods. [ABSTRACT FROM AUTHOR]

In image forensics, quantization step estimation plays a crucial role in revealing the JPEG compression history. This study focuses on the estimation for images that have been JPEG compressed and re-saved in lossless formats. Several effective methods have been developed, whereas the performance on small sized images still needs to be improved. In this paper, a novel JPEG quantization step estimation method with low complexity and high efficiency is proposed. First, for a given decompressed image, as the special shape of its DCT coefficients distribution is highly related to the quantization step, a function of the candidate step taking a similar shape is designed. Then, the quantization step is determined as the candidate leading to the maximum response of the designed function on the probability density function of DCT coefficients. The relation between the maximum response and the quantization step is verified mathematically, which provides a theoretical basis for the proposed method. In addition, to further improve the estimation performance, two fine adjustment procedures are adopted. Experimental results demonstrate that the proposed method outperforms some state-of-the-art works, especially on small sized images. [ABSTRACT FROM AUTHOR]

Homography estimation is an important task in computer vision applications, such as image stitching, video stabilization, and camera calibration. Traditional homography estimation methods heavily depend on the quantity and distribution of feature correspondences, leading to poor robustness in low-texture scenes. The learning solutions, on the contrary, try to learn robust deep features but demonstrate unsatisfying performance in the scenes with low overlap rates. In this paper, we address these two problems simultaneously by designing a contextual correlation layer (CCL). The CCL can efficiently capture the long-range correlation within feature maps and can be flexibly used in a learning framework. In addition, considering that a single homography can not represent the complex spatial transformation in depth-varying images with parallax, we propose to predict multi-grid homography from global to local. Moreover, we equip our network with a depth perception capability, by introducing a novel depth-aware shape-preserved loss. Extensive experiments demonstrate the superiority of our method over state-of-the-art solutions in the synthetic benchmark dataset and real-world dataset. The codes and models will be available at https://github.com/nie-lang/Multi-Grid-Deep-Homography. [ABSTRACT FROM AUTHOR]

The occurrence of motion blur along with defocus blur is a common phenomena in natural images. Usually, these blurs are spatially varying in nature for any general image and estimation of one type of blur is affected by presence of other. In this paper, we propose a novel method to estimate the concurrent defocus and motion blurs in a single image. Unlike the recent methods, which perform well only on simulated conditions or in presence of single type of blur, proposed method works well for real images as well as for compressed images. In this paper, we consider only commonly associated motion and defocus blurs for analysis. Decoupling of motion and defocus blur provides a fundamental tool that can be used for various analysis and applications. [ABSTRACT FROM AUTHOR]

In this paper, we propose an iterative feature matching framework for self-supervised depth estimation in indoor scenes. Conventional methods usually leverage the structure-from-motion supervision to help the photometric optimization escape from the local minima, which have complex ego-motion and large regions with non-texture or repeated-texture. However, the supervision is limited as the reconstruction is usually sparse. To address this, we propose an iterative feature matching framework called IFMNet to jointly learn depths and search for correspondences. With the predicted depths from the previous iteration, we present an online optimized grid searching algorithm to find more accurate correspondences. Given these new correspondences, we compute the triangulated depths and improve the depth network with adaptive bin-wise online hard example mining. Experimental results on the NYU Depth V2 and SceneNet datasets verify the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

A noise-suppressing low-light image enhancement approach is proposed in this paper based on the extent of exposedness at each image pixel. To this end, a progressive, structure-aware exposedness estimation procedure is presented that quantifies local and global exposedness. These exposedness values are leveraged to produce a locally smooth pixel-level map that signifies the required degrees of enhancement at image pixels. This map is subsequently used in an enhancement function, which satisfies a few important properties, to generate the enhanced image. Before the enhancement, inherent noise in the low-light image is diminished employing a detail-preserving, low gradient magnitude suppression method. Subjective and quantitative analysis of results on a wide variety of natural and synthetically generated low-light images from standard databases using PSNR, iRSE, SSIM, and measures of perceptual quality, natural image statistics and brightness preservation suggests that our approach in general outperforms the state-of-the-art. Ablation studies and further experiments show the importance of a few components of our approach, and that our approach is computationally fast. [ABSTRACT FROM AUTHOR]

In reversible data hiding (RDH) schemes, how to select those smooth pixels, pixel pairs or pixel blocks in order to improve performance is an important issue. For pixel-value-ordering (PVO) based RDH schemes, two existing techniques appear to be defective since only two reference pixels in a block or the right and bottom neighbors of a block are exploited as the context for block selection, and their performance might be only adequate for those smooth images. For rough images, the embedding performance could be affected. In this paper, an efficient block selection method by computing a block’s smoothness with a full-enclosing context (FEC) way is proposed. Obtained results show that the proposed FEC strategy can better estimate a block’s smoothness for PVO-based schemes. Furthermore, a more scalable pairing way is presented for the recently reported location-based PVO predictor. The proposed PVO scheme can be implemented by dividing the cover image and embedding bits into two different types of blocks, respectively. Experimental results show that the marked images by the proposed two-stage PVO scheme have higher visual quality, e.g., the average PSNR for the Kodak image database is 63.31 dB after embedding 10,000 bits, and the gain is 0.16 dB against the best result in the literature. Compared with some state-of-the-art RDH works, the superiority of the proposed algorithm has been verified in extensive experiments. [ABSTRACT FROM AUTHOR]

Efficient real-time disparity estimation is critical for the application of stereo vision systems in various areas. Recently, stereo network based on coarse-to-fine method has largely relieved the memory constraints and speed limitations of large-scale network models. Nevertheless, all of the previous coarse-to-fine designs employ constant offsets and three or more stages to progressively refine the coarse disparity map, still resulting in unsatisfactory computation accuracy and inference time when deployed on mobile devices. This paper claims that the coarse matching errors can be corrected efficiently with fewer stages as long as more accurate disparity candidates can be provided. Therefore, we propose a dynamic offset prediction module to meet different correction requirements of diverse objects and design an efficient two-stage framework. In addition, a disparity-independent convolution is proposed to regularize the compact cost volume efficiently and further improve the overall performance. The disparity quality and efficiency of various stereo networks are evaluated on multiple datasets and platforms. Evaluation results demonstrate that, the disparity error rate of the proposed network achieves 2.66% and 2.71% on KITTI 2012 and 2015 test sets respectively, where the computation speed is $2\times $ faster than the state-of-the-art lightweight models on high-end and source-constrained GPUs. [ABSTRACT FROM AUTHOR]

Depth estimation from a camera is an important task for 3D perception. Recently, without using the labeled ground truth of depth map, a self-supervised deep learning network can use relative pose to synthesize the target image from the reference image, and the photometric error between synthesized reference image and real one is used as self-supervisory signal. In this paper, we propose a novel self-supervised depth estimation network, which takes advantage of the quadtree constraint to optimize the depth estimation network. Based on the quadtree constraint, the photometric loss and depth loss of quadtree are proposed. In order to solve the problem that multiple depth values in repeated structures and uniform texture regions can cause relatively low photometric loss, we use quadtree-based photometric loss, which calculates the averaged photometric loss in quadtree blocks instead of the pixel-wise loss. For the problem of imbalanced depth distribution, we use quadtree depth loss, which constrains the depth inconsistency within quadtree blocks. The depth estimation network is composed of deep fusion module and cross-layer feature fusion module, which can better extract the feature information of RGB image and sparse keypoints depths, and makes full use of the detail information of the shallow feature map and the semantic information of the deep feature map to enrich the feature information extraction. Experimental results demonstrate that our method outperforms the state-of-the-art approaches of depth estimation. [ABSTRACT FROM AUTHOR]

Blind noise-level estimation (NLE) is a fundamental issue in digital image processing. This paper provides a noise-level estimator for additive white Gaussian noise and multiplicative Gaussian noise using principal texture patches (PTPs). The proposed algorithm first identifies the principal texture of the noisy image by using the principal component analysis, and then, it chooses PTPs to calculate the noise level. The major contributions of this paper toward addressing the challenges in the NLE literature include: 1) analyzing the nonlinear relationship between the smallest eigenvalue, true noise level, number of chosen patches, and block size; 2) extracting PTPs; and 3) computing noise level using PTPs. The abundant experimental results show that the proposed method works well for various natural images over a large range of noise levels and it performs well for multiplicative noise. Compared with some state-of-the-art approaches, our method has the best performance and a faster running speed. [ABSTRACT FROM AUTHOR]

Depth estimation is crucial in several computer vision applications, and a recent trend in this field aims at inferring such a cue from a single camera. Unfortunately, despite the compelling results achieved, state-of-the-art monocular depth estimation methods are computationally demanding, thus precluding their practical deployment in several application contexts characterized by low-power constraints. Therefore, in this paper, we propose a lightweight Convolutional Neural Network based on a shallow pyramidal architecture, referred to as $\mu $ PyD-Net, enabling monocular depth estimation on microcontrollers. The network is trained in a peculiar self-supervised manner leveraging proxy labels obtained through a traditional stereo algorithm. Moreover, we propose optimization strategies aimed at performing computations with quantized 8-bit data and map the high-level description of the network to low-level layers optimized for the target microcontroller architecture. Exhaustive experimental results on standard datasets and an in-depth evaluation with a device belonging to the popular Arm Cortex-M family confirm that obtaining sufficiently accurate monocular depth estimation on microcontrollers is feasible. To the best of our knowledge, our proposal is the first one enabling such remarkable achievement, paving the way for the deployment of monocular depth cues onto the tiny end-nodes of distributed sensor networks. [ABSTRACT FROM AUTHOR]

Inter prediction is a crucial part of hybrid video coding frameworks, utilized to exploit the temporal redundancy in video sequences and improve the coding performance. During inter prediction, a predicted block is typically derived from reference pictures using motion estimation and motion compensation. To improve the coding performance of inter prediction, a neural network based enhancement to inter prediction (NNIP) is proposed in this paper. NNIP is composed of three networks, namely residue estimation network, combination network, and deep refinement network. Specifically, first, a residue estimation network is designed to estimate the residue between current block and its predicted block using their available spatial neighbors. Second, the feature maps of the estimated residue and the predicted block are extracted and concatenated in a combination network. Finally, the concatenated feature maps are fed into a deep refinement network to generate a refined residue, which is added back to the predicted block to derive a more accurate predicted block. NNIP is integrated in HEVC to evaluate its efficiency. The experimental results demonstrate that NNIP can achieve 4.6%, 3.0%, and 2.7% BD-rate reduction on average under LDP, LDB, and RA configurations compared to HEVC. [ABSTRACT FROM AUTHOR]

Many unsupervised approaches have been proposed recently for the video-based re-identification problem since annotations of samples across cameras are time-consuming. However, higher-order relationships across the entire camera network are ignored by these methods, leading to contradictory outputs when matching results from different camera pairs are combined. In this paper, we address the problem of unsupervised video-based re-identification by proposing a consistent cross-view matching (CCM) framework, in which global camera network constraints are exploited to guarantee the matched pairs are with consistency. Specifically, we first propose to utilize the first neighbor of each sample to discover relations among samples and find the groups in each camera. Additionally, a cross-view matching strategy followed by global camera network constraints is proposed to explore the matching relationships across the entire camera network. Finally, we learn metric models for camera pairs progressively by alternatively mining consistent cross-view matching pairs and updating metric models using these obtained matches. Rigorous experiments on two widely-used benchmarks for video re-identification demonstrate the superiority of the proposed method over current state-of-the-art unsupervised methods; for example, on the MARS dataset, our method achieves an improvement of 4.2% over unsupervised methods, and even 2.5% over one-shot supervision-based methods for rank-1 accuracy. [ABSTRACT FROM AUTHOR]

Haze caused by atmospheric scattering and absorption would severely affect scene visibility of an image. Thus, image dehazing for haze removal has been widely studied in the literature. Within a hazy image, haze is not confined in a small local patch/position, while widely diffusing in a whole image. Under this circumstance, global context is a crucial factor in the success of dehazing, which was seldom investigated in existing dehazing algorithms. In the literature, the global context (GC) block has been designed to learn point-wise long-range dependencies of an image for global context modeling; however, patch-wise long-range dependencies were ignored. To image dehazing, patch-wise long-range dependencies should be highlighted to cooperate with patch-wise operations of image dehazing. In this paper, we first extend the point-wise GC into a Pyramid Global Context (PGC), which is a multi-scale GC, after undergoing the pyramid pooling. Thus, patch-wise long-range dependencies can be explored by the PGC. Then, the proposed PGC is plugged into a U-Net, getting an attentive U-Net. Further, the attentive U-Net is optimized by importing ResNet’s shortcut connection and dilated convolution. Thus, the finalized dehazing model can explore both long-range and patch-wise context dependencies for global context modeling, which is crucial for image dehazing. The extensive experiments on synthetic databases and real-world hazy images demonstrate the superiority of our model over other representative state-of-the-art models from both quantitative and qualitative comparisons. [ABSTRACT FROM AUTHOR]

The success of convolutional neural network (CNN) based single image dynamic scene blind deblurring (SIDSBD) methods mainly stems from the multi-scale/multi-patch model and the designs of the encoder-decoder architecture, and the residual block structure, which make different contributions to SIDSBD. In this paper, we further exploit the advantages of the multi-scale model, the encoder-decoder module, and the residual block structure, respectively, and propose a novel multi-scale channel attention network (MSCAN) for effective single image dynamic scene blind deblurring. Different from existing multi-scale models, in our proposed network, each scale consists of multiple levels, in which a novel spatial pyramid pooling channel attention (SPPCA) strategy is proposed to adaptively rescale the channel-wise features by using both the global and local feature statistics for more powerful network representation. Extensive experiments on both the synthetic benchmark datasets and the real blurred images show that our method can produce better deblurring results than the state-of-the-art SIDSBD methods in terms of both qualitative evaluation and quantitative metrics. [ABSTRACT FROM AUTHOR]

Disparity estimation is an essential task taking part in many light-field applications. Due to the complexity of algorithms and high dimensional property of light-field data, performing this task involves a significant computational effort and results in very long processing time on CPU. Graphics processing units (GPUs), which is capable of massively parallel processing, is a promising solution to cover the computation requirement and speed up the task. In this paper, we develop a GPU-accelerated approach for light-field disparity estimation using a variational computation framework (GVLD). Our algorithm combines the intrinsic sub-pixel precision of variational formulation and the effectiveness of weighted median filtering to produce a highly accurate solution. The proposed algorithm is fully parallelized and optimized for the implementation using the OpenCL framework. An intensive evaluation including a quantitative comparison to related works and a detailed analysis of the proposed approach’s performance is presented. Experimental results demonstrate our superior performance compared to state-of-the-art approaches. The proposed approach is 10+ times faster than other approaches running on a similar GPU platform and provides the most accurate solution among optimization-based approaches. Compared to the implementation running on CPU, our GPU-accelerated method achieves up to $365\times $ speed up. [ABSTRACT FROM AUTHOR]

The function of an image dehazing algorithm restores a hazy image back to a clear one. This technique can be equipped in Advanced Driver Assistance Systems (ADAS) to further improve driving safety. Advanced display resolutions and high frame rates are critical to improve the reliability of ADAS. However, they cause a serious limitation on the hardware realization of an image dehazing algorithm. In this paper, an algorithm and architecture design of a hardware-efficient image dehazing engine are proposed. This dehazing algorithm comprises two core techniques: a low-complexity airlight estimation (LAE) and an independent transmission rate estimation (ITRE). LAE can efficiently predict the airlight without massive sorting processes and a computationally intensive filtering. Moreover, in its hardware architecture, a preluded airlight estimation is proposed to yield an efficient data scheduling. Being independent of the airlight, ITRE is able to solely predict transmission rates. This enables LAE and ITRE to be concurrently processed, improving the throughput of their hardware architectures. Compared with other sophisticated studies, this study demonstrates a competitive performance in peak signal-to-noise ratio, structural similarity, contrast distortion, and computational efficiency. The hardware architecture is realized using Taiwan Semiconductor Manufacturing Company CMOS 0.18- $\mu \text{m}$ technology, and logic gate count is 32.6 K. Its working frequency is 200 MHz with a power consumption of 16.5 mW. The maximum throughput is as high as 600 megapixels/s, which can support $4\,096\times 2\,160$ at 60 frames/s. The experimental results indicate that this paper demonstrates superior energy and area efficiencies. [ABSTRACT FROM AUTHOR]

In many practical engineering applications, the number of actions that have been finished should be known, particularly for an untrimmed video sequence that includes an event with a series of actions, it is important to know the number of actions that have been finished. In this paper, we termed this process as visual event progress estimation (EPE). However, the research related to this problem is few in the research community. To solve this problem, a visual human action analysis-based framework, namely one-shot simultaneously action detection and identification (SADI)-EPE, is presented in this paper. The visual EPE is modeled as an online one-shot learning-based problem; sliding window and attention-based bag of key poses formulate our framework. Unlike most of the action analysis methods relying on a number of training data of some predefined classes, our method can realize SADI for any event if one sample of the event is given, which makes it feasible for practical applications. At the same time, not only SADI but also the progress estimation of the event can be realized by our algorithm. In terms of methodology, the key pose is defined by an invariant pose descriptor from skeletal data and silhouette data. Moreover, in order to extract representative and discriminative poses from one training sample, we present a new bidirectional $k$ NN-based attention weighted key pose selection method, which can filter the unrelated actions and model different importance of various key poses. In addition, an attention-based multi-modal fusion scheme, which addresses the difficulty of high-dimensional features and few training samples, is proposed to augment the performance of our algorithm. Finally, we propose an evaluation criterion for the estimation problem. Extensive results demonstrated the efficacy of our proposed framework. [ABSTRACT FROM AUTHOR]

Quality assessment of omnidirectional images has become increasingly urgent due to the rapid growth of virtual reality applications. Different from traditional 2D images and videos, omnidirectional contents can provide consumers with freely changeable viewports and a larger field of view covering the 360° × 180° spherical surface, which makes the objective quality assessment of omnidirectional images more challenging. In this paper, motivated by the characteristics of the human vision system (HVS) and the viewing process of omnidirectional contents, we propose a novel Viewport oriented Graph Convolution Network (VGCN) for blind omnidirectional image quality assessment (IQA). Generally, observers tend to give the subjective rating of a 360-degree image after passing and aggregating different viewports information when browsing the spherical scenery. Therefore, in order to model the mutual dependency of viewports in the omnidirectional image, we build a spatial viewport graph. Specifically, the graph nodes are first defined with selected viewports with higher probabilities to be seen, which is inspired by the HVS that human beings are more sensitive to structural information. Then, these nodes are connected by spatial relations to capture interactions among them. Finally, reasoning on the proposed graph is performed via graph convolutional networks. Moreover, we simultaneously obtain global quality using the entire omnidirectional image without viewport sampling to boost the performance according to the viewing experience. Experimental results demonstrate that our proposed model outperforms state-of-the-art full-reference and no-reference IQA metrics on two public omnidirectional IQA databases. [ABSTRACT FROM AUTHOR]

Quantization plays a pivotal role in JPEG compression with respect to the tradeoff between image fidelity and storage size, and the blind estimation of quantization parameters has attracted considerable interest in the fields of image steganalysis and forensics. Existing estimation methods have made great progress, but they usually suffer a sharp decline in accuracy when addressing small-size JPEG decompressed bitmaps due to the insufficiency of coefficients. Aiming to alleviate this issue, this paper proposes a generic clustering-based framework to improve the performance of the existing methods. The core idea is to gather as many coefficients as possible by clustering subbands before feeding them into a step estimator. The proposed framework is implemented using hierarchical clustering with two kinds of histogram-like features. Extensive experiments are conducted to validate the effectiveness of the proposed framework on a variety of images of different sizes and quality factors, and the results show that notable improvements can be achieved. In addition to quantization step estimation, we believe the idea behind the proposed framework might provide inspiration for other forensic tasks to alleviate their performance issues induced by sample insufficiency. [ABSTRACT FROM AUTHOR]

In this paper, we propose an adaptive region proposal scheme with feature channel regularization to facilitate robust object tracking. We consider tracking as a linear regression problem and an ensemble of correlation filters is trained on-line to distinguish the foreground target from the background. Further, we integrate adaptively learned region proposals into an enhanced two-stream tracking framework based on correlation filters. For the tracking stream, we learn two-stage cascade correlation filters on deep convolutional features to ensure competitive tracking performance. For the detection stream, we employ adaptive region proposals, which are effective in recovering target objects from tracking failures caused by heavy occlusion or out-of-view movement. In contrast to traditional tracking-by-detection methods using random samples or sliding windows, we perform target re-detection over adaptively learned region proposals. Since region proposals naturally take the objectness information into account, we show that the proposed adaptive region proposals can handle the challenging scale estimation problem as well. In addition, we observe the channel redundancy and noisy of feature representation, especially for the convolutional features. Thus, we apply a channel regularization to the correlation filter learning. Extensive experimental validations on OTB, VOT and UAV-123 datasets demonstrate that the proposed method performs favorably against state-of-the-art tracking algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new R/D estimation method for intra-predictive coding with deep block partitioning structures. In our proposed R/D prediction, we adopt a particle filtering based prediction (PFP) to precisely predict intermediate R/D estimates for the next frame in a stochastic manner, which helps increasing the prediction accuracy of fast changing R/D values. Then, based on the intermediate R/D estimates by PFP, we infer an optimal model parameter of the TC’s probability density function (pdf) via convex optimization. We found that the proposed method brings about more stable R/D estimation performance thanks to both the improved prediction accuracy using the PFP for abrupt changes in true R/D values and the precise estimation of the optimal model parameter. Experimental results show that our method significantly reduces the normalized root mean square error from average 3.17 to 0.79 (74.90% reduction) for rate and from average 2.32 to 0.82 (64.61% reduction) for distortion, compared to the state-of-the art method. [ABSTRACT FROM AUTHOR]

In this paper, the problem of human action recognition using 3D reconstruction data is deeply investigated. 3D reconstruction techniques are employed for addressing two of the most challenging issues related to human action recognition in the general case, namely view variance (i.e., when the same action is observed from different viewpoints) and the presence of (self-) occlusions (i.e., when for a given point of view a body part of an individual conceals another body part of the same or another subject). The main contributions of this paper are summarized as follows. The first one is a detailed examination of the use of 3D reconstruction data for performing human action recognition. The latter includes the introduction of appropriate local/global flow/shape descriptors, extensive experiments in challenging publicly available datasets and exhaustive comparisons with state-of-art approaches. The second one is a new local-level 3D flow descriptor, which incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every local neighborhood. The third one is a new global-level 3D flow descriptor that efficiently encodes the global motion characteristics in a compact way. The fourth one is a novel global temporal-shape descriptor that extends the notion of 3D shape descriptions for action recognition, by incorporating the temporal dimension. The proposed descriptor efficiently addresses the inherent problems of temporal alignment and compact representation, while also being robust in the presence of noise (compared with similar tracking-based methods of the literature). Overall, this paper significantly improves the state-of-art performance and introduces new research directions in the field of 3D action recognition, following the recent development and wide-spread use of portable, affordable, high-quality and accurate motion capturing devices (e.g., Microsoft Kinect). [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation aims to generalize a model from the label-rich source domain to the unlabeled target domain. Existing works mainly focus on aligning the global distribution statistics between source and target domains. However, they neglect distractions from the unexpected noisy samples in domain distribution estimation, leading to domain misalignment or even negative transfer. In this paper, we present an importance sampling method for domain adaptation (ISDA), to measure sample contributions according to their “informative” levels. In particular, informative samples, as well as outliers, can be effectively modeled using feature-norm and prediction entropy of the network. The importance of information is further formulated as the importance sampling losses in features and label spaces. In this way, the proposed model mitigates the noisy outliers while enhancing the important samples during domain alignment. In addition, our model is easy to implement yet effective, and it does not introduce any extra parameters. Extensive experiments on several benchmark datasets show that our method outperforms state-of-the-art methods under both the standard and partial domain adaptation settings. [ABSTRACT FROM AUTHOR]

In recent years, with the improvements in machine learning, image forensics has made considerable progress in detecting editing manipulations. This progress also raises more questions in image forensics research, such as can the parameters applied in a manipulation be estimated. Many parameter estimation works have already been performed. However, most of these works are based on mathematical analyses. In this paper, we attempt to solve a particular parameter estimation problem from a different aspect. Specifically, a new convolutional neural network (CNN) model is proposed to estimate the resampling rate for resampled images regardless of whether the image is upscaled or downscaled. This model features an original layer to generate a measurable energy map toward the estimation of resampling rate (METEOR). The METEOR layer is demonstrated to be an outstanding method that can assist in enhancing the estimation performance of the CNN. Furthermore, the METEOR layer can also increase the robustness of the CNN against JPEG compression, which makes it extremely important in realistic application scenarios. Our work has verified that machine learning, particularly CNNs, with proper optimization can also be refined to adapt to parameter estimation in digital forensics with excellent performance and robustness. [ABSTRACT FROM AUTHOR]

Recently discriminative correlation filter (DCF) based methods have gained much popularity for their excellent performance and high efficiency. However, most of them perform poorly in long-term tracking as they are not equipped with an effective mechanism to evaluate the quality of tracking results and correct tracking errors. To resolve such issue, this paper proposes a long-term tracking method, which consists of two components, including tracking-by-detection and re-detection. The tracking-by-detection part is built upon the DCF framework by incorporating a contour constraint map, which could identify non-target samples and refine the tracking results efficiently in presence of some challenging situations, such as deformation and occlusion. Benefited by our proposed re-detection strategy, the missing/occluded target could be captured immediately after it reappears. Moreover, a re-detected result is allowed to replace the original tracking result only when it owns a performance gap against the original one, which can reduce the risk of wrong substitution and well enhance the long-term tracking robustness. Extensive experiments on OTB2015, Temple-Color, UAV20L and VOT-LT2018 show that the proposed long-term tracking method outperforms state-of-the-art hand-crafted based methods, and even some deep learning based methods. [ABSTRACT FROM AUTHOR]

Existing light field depth map estimation approaches only utilize partial angular views in occlusion areas and local spatial dependencies in the optimization. This paper proposes a novel two-stage light field depth estimation method via graph spectral analysis to exploit the complete correlations and dependencies within angular patches and spatial images. The initial depth map estimation leverages the undirected graph to jointly consider occluded and unoccluded views within each angular patch. The estimated depth minimizes the structural incoherence of its corresponding angular patch with the focused one by evaluating the highest graph frequency component. Subsequently, depth map refinement optimizes the initial depth map with the color consistency and smoothness formulated by weighted adjacency matrix. The structural constraints are efficiently employed using low-pass graph filtering with Chebyshev polynomial approximation. Experimental results demonstrate that the proposed method improves the depth map estimation, especially in the edge regions. [ABSTRACT FROM AUTHOR]

Degraded visibility and geometrical distortion typically make the underwater vision more intractable than open air vision, which impedes the development of underwater-related machine vision and robotic perception. Therefore, this paper addresses the problem of joint underwater depth estimation and color correction from monocular underwater images, which aims at enjoying the mutual benefits between these two related tasks from a multi-task perspective. Our core ideas lie in our new deep learning architecture. Due to the lack of effective underwater training data, and the weak generalization to the real-world underwater images trained on synthetic data, we consider the problem from a novel perspective of style-level and feature-level adaptation, and propose an unsupervised adaptation network to deal with the joint learning problem. Specifically, a style adaptation network (SAN) is first proposed to learn a style-level transformation to adapt in-air images to the style of underwater domain. Then, we formulate a task network (TN) to jointly estimate the scene depth and correct the color from a single underwater image by learning domain-invariant representations. The whole framework can be trained end-to-end in an adversarial learning manner. Extensive experiments are conducted under air-to-water domain adaptation settings. We show that the proposed method performs favorably against state-of-the-art methods in both depth estimation and color correction tasks. [ABSTRACT FROM AUTHOR]

Image dehazing is a crucial image processing step for outdoor vision systems. However, images recovered through conventional image dehazing methods that use either haze-relevant priors or heuristic cues to estimate transmission maps may not lead to sufficiently accurate haze removal from single images. The most commonly observed effects are darkened and brightened artifacts on some areas of the recovered images, which cause considerable loss of fidelity, brightness, and sharpness. This paper develops a variational image dehazing method on the basis of a color-transfer image dehazing model that is superior to conventional image dehazing methods. By creating a color-transfer image dehazing model to remove haze obscuration and acquire information regarding the coefficients of the model by using the devised convolutional neural network-based deep framework as a supervised learning strategy, an image fidelity, brightness, and sharpness can be effectively restored. The experimental results verify through quantitative and qualitative evaluations of either synthesized or real haze images, and the proposed method outperforms existing single image dehazing methods. [ABSTRACT FROM AUTHOR]

This paper proposes a novel approach for crowd counting in low to high density scenarios in static images. Current approaches cannot handle huge crowd diversity well and thus perform poorly in extreme cases, where the crowd density in different regions of an image is either too low or too high, leading to crowd underestimation or overestimation. The proposed solution is based on the observation that detecting and handling such extreme cases in a specialized way leads to better crowd estimation. Additionally, existing methods find it hard to differentiate between the actual crowd and the cluttered background regions, resulting in further count overestimation. To address these issues, we propose a simple yet effective modular approach, where an input image is first subdivided into fixed-size patches and then fed to a four-way classification module labeling each image patch as low, medium, high-dense or no-crowd. This module also provides a count for each label, which is then analyzed via a specifically devised novel decision module to decide whether the image belongs to any of the two extreme cases (very low or very high density) or a normal case. Images, specified as high- or low-density extreme or a normal case, pass through dedicated zooming or normal patch-making blocks respectively before routing to the regressor in the form of fixed-size patches for crowd estimate. Extensive experimental evaluations demonstrate that the proposed approach outperforms the state-of-the-art methods on four benchmarks under most of the evaluation criteria. [ABSTRACT FROM AUTHOR]

Recently, optical flow estimation benefits greatly from deep learning based techniques. Most approaches use encoder-decoder architecture (U-Net) or spatial pyramid network (SPN) to learn optical flow. Both U-Net and SPN can extract multi-scale features and can predict optical flow directly. However, existing networks ignore to exploit the global information among channel features and inter-spatial relationship of features. In this paper, we propose a dual self-attention pyramid network, which adaptively integrates local features with their global dependencies and focuses on important features and suppresses unimportant features. Specifically, we introduce two types of attention modules into SPN, which emphasizes meaningful features along channel and spatial axes. The channel attention can adaptively re-weight channel-wise features by considering interdependencies among channels. Moreover, the spatial attention can utilize global contextual information to emphasize or suppress features in different spatial locations. In addition, two attention modules are embedded into each pyramidal level, which can refine features at different scale. We evaluate our method on MPI-Sintel and KITTI. The experimental results show that using the dual self-attention module can improve the representation power of network and further increase the accuracy of optical flow estimation. [ABSTRACT FROM AUTHOR]

Most existing algorithms for depth estimation from single monocular images need large quantities of metric ground-truth depths for supervised learning. We show that relative depth can be an informative cue for metric depth estimation and can be easily obtained from vast stereo videos. Acquiring metric depths from stereo videos are sometimes impracticable due to the absence of camera parameters. In this paper, we propose to improve the performance of metric depth estimation with relative depths collected from stereo movie videos using existing stereo matching algorithm. We introduce a new “relative depth in stereo” (RDIS) dataset densely labeled with relative depths. We first pretrain a ResNet model on our RDIS dataset. Then, we finetune the model on RGB-D datasets with metric ground-truth depths. During our finetuning, we formulate depth estimation as a classification task. This re-formulation scheme enables us to obtain the confidence of a depth prediction in the form of probability distribution. With this confidence, we propose an information gain loss to make use of the predictions that are close to ground-truth. We evaluate our approach on both indoor and outdoor benchmark RGB-D datasets and achieve the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Reversible image watermarking is a technique that allows the cover image to remain unmodified after watermark extraction. Prediction error expansion-based schemes are currently the most efficient and widely used class of reversible image watermarking techniques. In this paper, first, we prove that the bounded capacity distortion minimization problem for prediction error expansion-based reversible watermarking schemes is NP-hard, and the corresponding decision version of the problem is NP-complete. Then, we prove that the dual problem of bounded distortion capacity maximization problem for prediction error expansion-based reversible watermarking schemes is NP-hard, and the corresponding decision problem is NP-complete. Furthermore, taking advantage of the integer linear programming formulations of the optimization problems, we find the optimal performance metric values for a given image, using concepts from the optimal linear prediction theory. Our technique allows the calculation of these performance metric limit without assuming any particular prediction scheme. The experimental results for several common benchmark images are consistent with the calculated performance limits validate our approach. [ABSTRACT FROM AUTHOR]

Blind image deblurring is a challenging problem in computer vision, which aims to restore both the blur kernel and the latent sharp image from only a blurry observation. Inspired by the prevalent self-example prior in image super-resolution, in this paper, we observe that a coarse enough image down-sampled from a blurry observation is approximately a low-resolution version of the latent sharp image. We prove this phenomenon theoretically and define the coarse enough image as a latent structure prior of the unknown sharp image. Starting from this prior, we propose to restore sharp images from the coarsest scale to the finest scale on a blurry image pyramid and progressively update the prior image using the newly restored sharp image. These coarse-to-fine priors are referred to as multi-scale latent structures (MSLSs). Leveraging the MSLS prior, our algorithm comprises two phases: 1) we first preliminarily restore sharp images in the coarse scales and 2) we then apply a refinement process in the finest scale to obtain the final deblurred image. In each scale, to achieve lower computational complexity, we alternately perform a sharp image reconstruction with fast local self-example matching, an accelerated kernel estimation with error compensation, and a fast non-blind image deblurring, instead of computing any computationally expensive non-convex priors. We further extend the proposed algorithm to solve more challenging non-uniform blind image deblurring problem. The extensive experiments demonstrate that our algorithm achieves the competitive results against the state-of-the-art methods with much faster running speed. [ABSTRACT FROM AUTHOR]

Entropy coding is a fundamental technology in video coding that removes statistical redundancy among syntax elements. In high efficiency video coding (HEVC), context-adaptive binary arithmetic coding (CABAC) is adopted as the primary entropy coding method. The CABAC consists of three steps: binarization, context modeling, and binary arithmetic coding. As the binarization processes and context models are both manually designed in CABAC, the probability of the syntax elements may not be estimated accurately, which restricts the coding efficiency of CABAC. To address the problem, we propose a convolutional neural network-based arithmetic coding (CNNAC) method and apply it to compress the syntax elements of the intra-predicted residues in HEVC. Instead of manually designing the binarization processes and context models, we propose directly estimating the probability distribution of the syntax elements with a convolutional neural network (CNN), as CNNs can adaptively build complex relationships between inputs and outputs by training with a lot of data. Then, the values of the syntax elements, together with their estimated probability distributions, are fed into a multi-level arithmetic codec to perform entropy coding. In this paper, we have utilized the CNNAC to code the syntax elements of the DC coefficient; the lowest frequency AC coefficient; the second, third, fourth, and fifth lowest frequency AC coefficients; and the position of the last non-zero coefficient in the HEVC intra-predicted residues. The experimental results show that our proposed method achieves up to 6.7% BD-rate reduction and an average of 4.7% BD-rate reduction compared to the HEVC anchor under all intra (AI) configuration. [ABSTRACT FROM AUTHOR]

Recently, region-based 3D video coding has been proposed. However, existing view synthesis distortion estimation (VSDE) methods are performed at the frame level. To guide the rate-distortion optimization process of region-based 3D video coding schemes, this paper proposes the first pixel-level VSDE (PL-VSDE) method. We first give the definition of the pixel-level view synthesis distortion. To estimate it, a backward prediction method is then developed, which starts from the pixels of interest (POIs) in the virtual view and finds their corresponding pixels in the reference view via a coarse-to-fine approach, denoted as coarse-to-fine backward prediction (CFBP) method. Additionally, the CFBP fully considers the details of 3D warping, the rounding operation and the warping competition in view synthesis, leading to improve accuracy of the prediction. Besides, a table-lookup method and a warping property are introduced to speed up the CFBP. After integrating the CFBP into the PL-VSDE, we can estimate the view synthesis distortion at the pixel level. Our method is carried out pixel-by-pixel independently, which is friendly for parallel processing. The experimental results demonstrate that our proposed method has significant advantages in both accuracy and efficiency compared with the state-of-the-art frame-level VSDE methods. [ABSTRACT FROM AUTHOR]