Search

The new generation of the high-efficiency video coding (HEVC) video coding standard has improved compression performance and brought considerable coding complexity. Therefore, reducing the perception redundancy of video to obtain a better compression effect is the new direction of video development at present. In this paper, the HEVC is improved and enhanced from the aspects of a video saliency algorithm based on an attention mechanism and a video compression algorithm based on perception priority. In terms of video saliency, this paper proposes a spatial saliency algorithm based on a convolutional neural network and a temporal saliency algorithm based on motion vector. The saliency algorithm can combine the motion estimation results of each block during the HEVC compression on the basis of convolutional neural network and carry out adaptive dynamic fusion of the two, to complete the saliency map of the input video. In the aspect of the video compression algorithm with perception priority, this paper proposes a more flexible QP selection method, which selects its corresponding QP according to the saliency value of CU. At the same time, we propose a new rate-distortion optimization algorithm, which integrates the current block’s saliency feature into the traditional rate-distortion calculation method, to guide the allocation of bits and achieve the purpose of perception priority. The experimental results proved the superiority of the proposed method over the state-of-the-art perceptual coding algorithms in terms of saliency detection and perceptual compression quality. [ABSTRACT FROM AUTHOR]

Screen content coding plays an important role in many applications. Conventional reference picture management (RPM) strategies developed for natural content may not work well for screen content. This is because many regions in screen content remain static for a long time, causing a lot of repetitive contents to stay in the decoded picture buffer. The repetitive contents are not conducive to inter prediction, but still occupy valuable memory. This paper proposes a diversity-based RPM scheme for screen content coding. The concept of diversity is introduced for the reference picture set (RPS) to help formulate the RPM problem. By maximizing the diversity of RPS, more potentially better predictions are provided. Better compression performance can then be achieved. Meanwhile, the proposed scheme is nonnormative and compatible with existing video coding standards, such as High Efficiency Video Coding. The experimental results show that, for low delay screen content coding, the bit saving of the proposed scheme is 4.9% on average and up to 13.7%, without increasing encoding time. [ABSTRACT FROM AUTHOR]

Rate control typically involves two steps: bit allocation and bitrate control. The bit allocation step can be implemented in various fashions depending on how many levels of allocation are desired and whether or not an optimal rate–distortion (R-D) performance is pursued. The bitrate control step has a simple aim in achieving the target bitrate as precisely as possible. In our recent research, we have developed a \lambda -domain rate control algorithm that is capable of controlling the bitrate precisely for High Efficiency Video Coding (HEVC). The initial research showed that the bitrate control in the \lambda -domain can be more precise than the conventional schemes. However, the simple bit allocation scheme adopted in this initial research is unable to achieve an optimal R-D performance reflecting the inherent R-D characteristics governed by the video content. In order to achieve an optimal R-D performance, the bit allocation algorithms need to be developed taking into account the video content of a given sequence. The key issue in deriving the video-content-guided optimal bit allocation algorithm is to build a suitable R-D model to characterize the R-D behavior of the video content. In this paper, to complement the R- \lambda model developed in our initial work, a D- \lambda model is properly constructed to complete a comprehensive framework of \lambda -domain R-D analysis. Based on this comprehensive \lambda -domain R-D analysis framework, a suite of optimal bit allocation algorithms are developed. In particular, we design both picture-level and basic-unit-level bit allocation algorithms based on the fundamental R-D optimization theory to take full advantage of the content-guided principles. The proposed algorithms are implemented in HEVC reference software, and the experimental results demonstrate that they can achieve an obvious R-D performance improvement with a smaller bitrate control error. The proposed bit allocation algorithms have already been adopted by the Joint Collaborative Team on Video Coding and integrated into the HEVC reference software. [ABSTRACT FROM PUBLISHER]

Flexible block partition and newly adopted intra coding tools bring significant performance improvement for next generation video coding, and meanwhile introduce non-negligible encoding complexity increment. This paper presents a scalable intra coding optimization (SICO) scheme for the third generation of audio video coding standard (AVS3). The complexity distribution and inheriting relationship among different partitioning, as well as intra prediction modes, are systematically analyzed. Subsequently, low-complexity algorithms are proposed, which could early exclude unlikely coding modes safely. In particular, a data-driven binary classifier is elegantly trained for the determination of coding unit partitioning. Moreover, the preliminary coding information can be exhaustively utilized for the mode decision in a low-cost manner. The proposed method provides scalable intra coding optimizing solutions, which is eligible to cater to various application scenarios. Experimental results show that the proposed method could achieve a wide range of encoding complexity reduction from 18% to 76% with moderate compression performance loss. One implementation of the proposed method has been adopted to the AVS3 reference software. [ABSTRACT FROM AUTHOR]

Busy life as well as the prevalence of infotainment is increasingly making people more occupied even during tasks that require serious attention. One such task is driving and at the same time getting involved in activities that may distract drivers cognitively from watching the road and cause fatal accidents. This paper presents a method that is capable of monitoring different types of distractions, such as talking and texting on cell phone, casual eating, and operating cabin equipment while driving, so that a driver can be assisted to remain cautious on the road. The proposed method automatically detects and tracks fiducial body parts of a driver from video captured by a camera mounted on the front windshield inside a vehicle. Relative distances between the tracking trajectories are used as features that represent actions of the driver. Then, the well-known kernel support vector machine is applied for recognizing a particular distraction from the features extracted from body parts. The proposed feature is also compared with previously employed features for tracking-based human action recognition schemes to substantiate its better result in terms of mean accuracy and robustness for distraction recognition. The effectiveness of the proposed method of distraction recognition is also analyzed with respect to tracking errors. [ABSTRACT FROM AUTHOR]

The classical solution to an underdetermined system of linear equations mainly has two opposite directions, which lead to either a large l2-norm sparse solution or a non-sparse minimum l2-norm solution. In this paper, we systematically show that by modifying the well-known basic matching pursuit algorithm originally proposed to identify the sparse solution, an alternative solution between the two classical ones could be obtained. The modified algorithm, termed as random matching pursuit (RMP), is then used to create a novel image watermarking framework. Compared to conventional systems, the security is substantially improved by the use of random over-complete dictionaries and the order parameter of RMP. Capacity can also be increased thanks to the transform with over-complete dictionaries that could expand signal dimension. Meanwhile, imperceptibility and robustness properties of the proposed design framework are not compromised. The classical spread spectrum and improved spread spectrum techniques are applied to the proposed framework for practical implementations. The novelty and effectiveness of the proposed systems are supported by rigorous performance analysis and experimental results using an image data set. This paper reveals the potential of using over-complete dictionaries in multimedia watermarking systems, which theoretically leads to the exploration of alternative candidates among the infinite solutions to underdetermined linear systems other than minimum l2-norm and sparse ones. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel rigid-body motion segmentation algorithm that uses randomized voting to assign high scores to correctly estimated models and low scores to wrongly estimated models. This algorithm is based on an epipolar geometrical representation of the camera motion, and computes scores using the distance between the feature point and the corresponding epipolar line. These scores are accumulated and utilized for motion segmentation. To evaluate the efficacy of our algorithm, we conduct a series of experiments using the Hopkins 155 data set and the UdG data set, which are representative test sets for rigid motion segmentation. Among several state-of-the-art data sets, our algorithm achieves the most accurate motion segmentation results and, in the presence of measurement noise, achieves comparable results to the other algorithms. Finally, we analyze why our motion segmentation algorithm works using probabilistic and theoretical analysis. [ABSTRACT FROM AUTHOR]

This paper presents a method to improve the traditional mixture of deformable part models (MDPM) method from the learning perspective. First, an object part configuration learning algorithm based on group sparsity constraint is introduced to automatically discover the object part number, size, and location. The algorithm imposes two additional regularization terms in addition to the standard hinge loss function. The first term focuses on automatic part selection and the second term focuses on automatic part placement. Second, this paper introduces an improved MDPM training framework. The framework applies a learned transformation to normalize the prediction score from each individual deformable part model (DPM) into a pseudo probability such that the partition of the entire object appearance feature space becomes less sensitive to the prior distributions of different DPMs. Finally, the two proposed improvements are combined and formulated under the expectation-maximization framework. We evaluate our method mainly using the PASCAL VOC2007 and VOC2010 detection benchmarks and show that the proposed learning algorithms could increase the detection mean AP score by 2.4% and 0.9%, respectively, on these two data sets when using the proposed part selection method and the training algorithm. We also present further in-depth analysis of the proposed algorithm in the experiments. [ABSTRACT FROM PUBLISHER]

This paper proposes a virtual 3D eyeglasses try-on system driven by a 2D Internet image of a human face wearing with a pair of eyeglasses. The main technical challenge of this system is the automatic 3D eyeglasses model reconstruction from the 2D glasses on a frontal human face. Against this challenge, this paper first proposes an eyeglasses segmentation method using a convolutional neural network-based parsing algorithm to label the glasses pixels, followed with a proposed symmetry-based level-set optimization algorithm to refine the contour of the eyeglasses. With the precisely extracted silhouette image, we take advantages of the smoothness and the symmetry priors of the eyeglasses, and propose a silhouette-based 3D deformation method to deform a 3D eyeglasses model selected from a predefined 3D model database. The obtained model is plausibly approximated to the input 2D eyeglasses after a texture mapping step. Finally, we develop a virtual try-on system to interactively synthesize the reconstructed eyeglasses on a moving target face in real time. The experimental results demonstrate the efficient and convincing virtual try-on performance of our approach and the commercial potential of our proposed system. [ABSTRACT FROM PUBLISHER]

As video data are occupying an increasingly more significant portion of global data traffic, video communication has become an indispensable component for most multimedia applications. As an enabling technology of video communication, although video coding is well standardized, the strategy to control video codec is highly customized to applications. The consistency of video quality is being paid more attention in many emerging applications. For example, in video surveillance, the quality of video frames should be stable in order to ensure the performance of machine intelligence algorithm, such as object detection precision. In this paper, we focus on achieving constant objective reconstruction quality in the process of video coding. To achieve certain rate-distortion performance, bit rate and distortion metric are usually modeled as a function of video content and control parameters of codec. For content modeling in existing work, there is still room for improvement, including the design of more efficient content feature, the compensation for assumption about constant RD characteristics among consecutive frames, and the adjustment of Lagrangian multiplier $\lambda $ according to content property. The main contributions of this paper are: 1) a robust content adaptive model for residual bit rate modeling based on content statistics called mean absolute partial transformed difference (MAPTD); 2) a content-related header bit rate modeling; 3) preprocessing scheme for robust content feature estimation at scene change; 4) a distortion model consistent with local content; and 5) content adaptive $\lambda $ determination. The experimental results show that our constant quality control strategy can achieve superior performance compared with the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Along with the rapid increase in the availability of high-quality video formats such as high definition (HD), ultra HD, and high dynamic range, a huge demand for data rates during their transmission has become inevitable. Consequently, the role of video compression techniques has become crucially important in the process of mitigating the data rate requirements. Even though the latest video codec high efficiency video coding (HEVC) has succeeded in significantly reducing the data rate compared to its immediate predecessor H.264/advanced video coding, the HEVC coded videos in the meantime have become even more vulnerable to network impairments. Therefore, it is equally important to assess the consumers’ perceived quality degradation prior to transmitting HEVC coded videos over an error-prone network, and to include error resilient features so as to minimize the adverse effects of those impairments. To this end, this paper proposes a probabilistic model which accurately predicts the overall distortion of the decoded video at the encoder followed by an accurate QP– $\lambda $ relationship which can be used in the rate-distortion optimization (RDO) process. During the derivation process of the probabilistic model, the impacts from the motion vectors, the pixels in the reference frames, and the clipping operations are accounted, and consequently, the model is capable of minimizing the prediction error to as low as 3.11%, whereas the state-of-the-art methods cannot reach below 20.08%, under identical conditions. Furthermore, the enhanced RDO process has resulted in 21.41%–43.59% improvement in the BD-rate compared to the state-of-the-art error resilient algorithms. [ABSTRACT FROM AUTHOR]

Compared with other applications in computer vision, convolutional neural networks (CNNs) have underperformed on pedestrian detection. A breakthrough was made very recently using sophisticated deep CNN (DCNN) models, with a number of handcrafted features or explicit occlusion handling mechanism. In this paper, we show that by reusing the convolutional feature maps of a DCNN model as image features to train an ensemble of boosted decision models, we are able to achieve the best reported accuracy without using specially designed learning algorithms. We empirically identify and disclose important implementation details. We also show that pixel labeling may be simply combined with a detector to boost the detection performance. By adding complementary handcrafted features such as optical flow, the DCNN-based detector can be further improved. We advance the state-of-the-art results by lowering the log-average miss rate from 11.7% to 8.9% on the Caltech data set and from 11.2% to 8.6% on the Inria data set. We also achieve a comparable result to state-of-the-art approaches on the KITTI data set. [ABSTRACT FROM AUTHOR]

3D video systems transmit depth maps in order to render synthesized views (SVs) at a receiver. To anticipate this purpose when processing a depth map, a sender-side depth processing algorithm (DPA), e.g. a depth encoder, can also render the SVs, compute their SV distortion (SVD), and adapt to it. This requires a low-complexity algorithm as computational resources are usually limited. We propose such an algorithm in this paper. First, we discuss a measure that relates a depth change to an SVD change using rendering. Then, we present an optimized process combining basic rendering steps, as warping, occlusion handling, interpolation, hole filling, and blending. Furthermore, we analyze which parts of an SV are affected by a depth change and modify the process to re-render only them. The resulting algorithm is significantly less complex than an unoptimized rendering-based variant and quantifies the SVD more accurately than existing estimation methods. The algorithm is used by the 3D-High Efficiency Video Coding reference software encoder as the main method for distortion computation and can also be used by other DPAs. [ABSTRACT FROM AUTHOR]

Optical flow calculation algorithms are hard to implement on the hardware level in real-time, due to their complexity and high computational load. Therefore, presented works in the literature focusing on the hardware implementation are limited. In this paper, we present a hierarchical block matching-based optical flow algorithm suitable for real-time hardware implementation. The algorithm estimates the initial optical flow with block matching and refines the vectors with local smoothness constraints in each level. We evaluate the proposed algorithm with novel data sets and provide results compared with the ground truth optical flow. Furthermore, we present a reconfigurable hardware architecture of the proposed algorithm for calculating the optical flow in real-time. The presented system can process $640\times 480$ resolution frames at 39 frames/s. [ABSTRACT FROM PUBLISHER]

A low-cost very large scale integration (VLSI) implementation of real-time correction of barrel distortion for video-endoscopic images is presented in this paper. The correcting mathematical model is based on least-squares estimation. To decrease the computing complexity, we use an odd-order polynomial to approximate the back-mapping expansion polynomial. By algebraic transformation, the approximated polynomial becomes a monomial form which can be solved by Hornor's algorithm. With the iterative characteristic of Hornor's algorithm, the hardware cost and memory requirement can be conserved by time multiplexed design. In addition, a simplified architecture of the linear interpolation is used to reduce more computing resource and silicon area. The VLSI architecture of this paper contains 13.9-K gates by using a 0.18 \mum CMOS process. Compared with some existing distortion correction techniques, this paper reduces at least 69% hardware cost and 75% memory requirement. [ABSTRACT FROM AUTHOR]

Detection of salient objects from images is gaining increasing research interest in recent years as it can substantially facilitate a wide range of content-based multimedia applications. Based on the assumption that foreground salient regions are distinctive within a certain context, most conventional approaches rely on a number of hand-designed features and their distinctiveness is measured using local or global contrast. Although these approaches have been shown to be effective in dealing with simple images, their limited capability may cause difficulties when dealing with more complicated images. This paper proposes a novel framework for saliency detection by first modeling the background and then separating salient objects from the background. We develop stacked denoising autoencoders with deep learning architectures to model the background where latent patterns are explored and more powerful representations of data are learned in an unsupervised and bottom-up manner. Afterward, we formulate the separation of salient objects from the background as a problem of measuring reconstruction residuals of deep autoencoders. Comprehensive evaluations of three benchmark datasets and comparisons with nine state-of-the-art algorithms demonstrate the superiority of this paper. [ABSTRACT FROM PUBLISHER]

This paper addresses the problem of semantic segmentation, where the possible class labels are from a predefined set. We exploit top-down guidance, i.e., the coarse localization of the objects and their class labels provided by object detectors. For each detected bounding box, figure–ground segmentation is performed and the final result is achieved by merging the figure–ground segmentations. The main idea of the proposed approach, which is presented in our preliminary work, is to reformulate the figure–ground segmentation problem as sparse reconstruction pursuing the object mask in a nonparametric manner. The latent segmentation mask should be coherent subject to sparse error caused by intra-category diversity; thus, the object mask is inferred by making use of sparse representations over the training set. To handle local spatial deformations, local patch-level masks are also considered and inferred by sparse representations over the spatially nearby patches. The sparse reconstruction coefficients and the latent mask are alternately optimized by applying the Lasso algorithm and the accelerated proximal gradient method. The proposed formulation results in a convex optimization problem; thus, the global optimal solution is achieved. In this paper, we provide theoretical analysis of the convergence and optimality. We also give an extended numerical analysis of the proposed algorithm and a comprehensive comparison with the related semantic segmentation methods on the challenging PASCAL visual object class object segmentation datasets and the Weizmann horse dataset. The experimental results demonstrate that the proposed algorithm achieves a competitive performance when compared with the state of the arts. [ABSTRACT FROM PUBLISHER]

In this paper, we address a super-resolution (SR) problem that constructs a high-resolution (HR) frame/image from a short sequence of low-resolution (LR) frames/images. It is well known that SR is a difficult problem, especially when the number of LR inputs is small. In particular, our previous work involving multisurface fitting-based SR exhibits relatively poor performance in the above case. To cope with this problem, we take advantage of nonlocal pixels to fit local surfaces. The pixels from nonlocal spatial–temporal positions are selected and weighted based on patch similarity and outlier removal. With this method, the fitted surfaces become more elaborate so that more details can be retrieved in SR results. Experiments demonstrate that the proposed method is very effective in producing HR frames through a small number of LR inputs when compared with some state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper presents the VLSI architecture and hardware implementation of a highly efficient deblocking filter (DBF) for High Efficiency Video Coding systems. In order to reduce the number of data accesses and thus to enhance the timing efficiency, novel data structures and memory access schemes for image pixels are proposed. Furthermore, a novel edge-fetching order is presented to strike a balance between the processing throughput and complexity. Based on the proposed structure and access pattern, a six-stage pipelined two-line DBF engine with low-latency data access sequence is designed, aiming to achieve high processing throughput while at the same time maintaining low complexity. The detailed storage structure and data access scheme are illustrated and VLSI architecture for the DBF engine is depicted in this paper. In addition, the proposed DBF is implemented using TSMC 90-nm standard cell library. The experimental results based on postlayout estimations show that the proposed design can achieve 60 frames/s for a frame resolution of $4096\times 2048$ pixels (ultra high definition resolution) assuming an operating frequency of 100 MHz. Moreover, this design occupies an area complexity of 466.5 kGE with a power consumption of 26.26 mW. In comparison with prior designs targeting similar system specification and throughput, the proposed design results in a significantly reduced area complexity. [ABSTRACT FROM PUBLISHER]

Over the past ten years, human re-identification has received increased attention from the computer vision research community. However, for the most part, these research papers are divorced from the context of how such algorithms would be used in a real-world system. This paper describes the unique opportunity our group of academic researchers had to design and deploy a human re-identification system in a demanding real-world environment: a busy airport. The system had to be designed from the ground up, including robust modules for real-time human detection and tracking, a distributed, low-latency software architecture, and a front-end user interface designed for a specific scenario. None of these issues are typically addressed in re-identification research papers, but all are critical to an effective system that end users would actually be willing to adopt. We detail the challenges of the real-world airport environment, the computer vision algorithms underlying our human detection and re-identification algorithms, our robust software architecture, and the ground-truthing system required to provide the training and validation data for the algorithms. Our initial results show that despite the challenges and constraints of the airport environment, the proposed system achieves very good performance while operating in real time. [ABSTRACT FROM PUBLISHER]

Image quality assessment is a critical problem for image compression, which can be utilized as a guidance for image compression and codec evaluation. In this paper, we propose a full reference image quality assessment (IQA) algorithm to measure the perceptual quality of compressed images. The proposed IQA model utilizes a data-driven transform, multi-stage Karhunen-Loeve Transform (MS-KLT), as a feature extractor to decompose both reference and compressed images into feature domain, where the importance of feature distortions in different spectral components to human visual system (HVS) is easy to distinguish. Accordingly, an efficient weighting strategy is proposed to reflect the importance of feature distortions based on the energy of transformed coefficients. Considering HVS characteristics, weighted spatial masking effect is derived from both local and global perspectives. In addition, to avoid influences of random noises, a local adaptive low-pass filtering process is applied as a pre-processing operation. Extensive experimental results on popular datasets show that our proposed method correlates better with the subjective results compared with the state-of-the-art algorithms. Moreover, the proposed method behaves more robustly compared with existing methods, and achieves top-ranking performance on different IQA datasets. [ABSTRACT FROM AUTHOR]

This paper presents an overview of the geometric partitioning mode (GPM) algorithm that is a part of the most recent Versatile Video Coding (VVC) standard. The GPM algorithm aims to increase the partitioning precision of moving objects using non-rectangular and asymmetric rectangular partitions on top of the conventional rectangular block partitioning structure of VVC. Novel features of GPM contributing to the increase in coding efficiency and the reduction in encoder and decoder complexity are detailed and analyzed in this paper. Evaluated with VVC test model version 8.0 under the joint video experts team common test conditions, experimental results show that the presented GPM algorithm provides luma Bjøntegaard Delta rate reduction of 0.70% for random access and of 1.55% for low delay with B slices configurations, with roughly 3% to 5% additional encoding time and negligible decoder runtime change. Furthermore, as GPM provides more precise partitions for the boundaries of the moving objects, an improvement of visual quality is seen in GPM coded sequences. [ABSTRACT FROM AUTHOR]

This paper presents an overview of the decoder-side motion vector refinement (DMVR) algorithm in the Versatile Video Coding (VVC) standard. The proposed DMVR algorithm aims to increase the prediction accuracy of the blocks coded in merge mode using the bilateral matching-based refinement method. Compared with previous decoder-side motion vector derivation approaches, the proposed method significantly increases the coding efficiency without signaling additional side information. Furthermore, the hardware implementation considerations of the DMVR design are particularly focused in this study. This paper details and analyzes the novel features of DMVR contributing to the increase in coding efficiency and the reduction in computational complexity and implementation difficulty. Experimental results based on the VVC test model version 8.0 demonstrate that average Bjøntegaard Delta rate savings of 0.80 % and 2.81 % are achieved for the “tool-off” and “tool-on” test configurations, respectively. Moreover, 4 % additional decoding time and negligible additional external memory bandwidth requirements of DMVR based on the common test conditions for VVC are reported. [ABSTRACT FROM AUTHOR]

Linear Regression (LR) is a popular and effective technique in pattern recognition area, which aims to find a transform matrix between source data and target data (usually label matrix). However, a binary zero-one label matrix may be too strict and inappropriate for regression. Besides, directly projecting source data to target data by one transform matrix may lose some intrinsic data information. To address these issues, this paper proposes a novel Pairwise Relations oriented Discriminative Regression (PRDR) method. In PRDR, the source data is regressed into a latent space instead of label space. To supervise the discriminative projection learning, the pairwise relations in source data space and label space are exploited in the latent space simultaneously. The pairwise label relations are transferred into the latent subspace by solving a distance-distance difference minimization problem, and the intraclass instance relations are also preserved in latent space. These two constraints ensure the pairwise similarity of data points after transformation which is beneficial for classification. By further enlarging the margins between true and false classes, PRDR is extended to a robust version, i.e., R-PRDR. An efficient algorithm is presented to solve the PRDR model. Extensive experiments on several popular image datasets demonstrate the effectiveness and efficiency of the proposed method compared with some state-of-the-art regression approaches. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) clustering is a challenging task due to the complex characteristics in HSI data, such as spatial-spectral structure, high-dimension, and large spectral variability. In this paper, we propose a novel deep spatial-spectral subspace clustering network (DS3C-Net), which explores spatial-spectral information via the multi-scale auto-encoder and collaborative constraint. Considering the structure correlations of HSI, the multi-scale auto-encoder is first designed to extract spatial-spectral features with different-scale pixel blocks which are selected as the inputs. Then, the collaborative constrained self-expressive layers are introduced between the encoder and decoder, to capture the self-expressive subspace structures. By designing a self-expressiveness similarity constraint, the proposed network is trained collaboratively, and the affinity matrices of the feature representation are learned in an end-to-end manner. Based on the affinity matrices, the spectral clustering algorithm is utilized to obtain the final HSI clustering result. Experimental results on three widely used hyperspectral image datasets demonstrate that the proposed method outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In many practical engineering applications, the number of actions that have been finished should be known, particularly for an untrimmed video sequence that includes an event with a series of actions, it is important to know the number of actions that have been finished. In this paper, we termed this process as visual event progress estimation (EPE). However, the research related to this problem is few in the research community. To solve this problem, a visual human action analysis-based framework, namely one-shot simultaneously action detection and identification (SADI)-EPE, is presented in this paper. The visual EPE is modeled as an online one-shot learning-based problem; sliding window and attention-based bag of key poses formulate our framework. Unlike most of the action analysis methods relying on a number of training data of some predefined classes, our method can realize SADI for any event if one sample of the event is given, which makes it feasible for practical applications. At the same time, not only SADI but also the progress estimation of the event can be realized by our algorithm. In terms of methodology, the key pose is defined by an invariant pose descriptor from skeletal data and silhouette data. Moreover, in order to extract representative and discriminative poses from one training sample, we present a new bidirectional $k$ NN-based attention weighted key pose selection method, which can filter the unrelated actions and model different importance of various key poses. In addition, an attention-based multi-modal fusion scheme, which addresses the difficulty of high-dimensional features and few training samples, is proposed to augment the performance of our algorithm. Finally, we propose an evaluation criterion for the estimation problem. Extensive results demonstrated the efficacy of our proposed framework. [ABSTRACT FROM AUTHOR]

The High Efficiency Video Coding (HEVC/H.265) standard achieves great improvement in compression efficiency over the widely used H.264/AVC standard at a cost of much higher complexity. When encoding videos using HEVC, the selection of the quantization parameter (QP) can significantly affect the coding efficiency. Typical algorithms for adaptive quantization employ fixed bitrate budgeting or fixed QP offsets for different frames and different blocks, without considering detailed input video characteristics, some of which might be captured using computer vision methods. The problem of determining adaptive settings of HEVC coding parameters has not been satisfactorily solved. In this paper, we proposed a Hadamard (“HAD”) energy-based optimized HEVC video encoder, in which HAD energy is used to measure the amount of residual information to be encoded in a block so as to determine the QP value for each block for a better coding efficiency. HAD energy is also used to expedite the time-consuming mode decision process and for a precise scene change detection to avoid flicker artifacts. Experiment using the widely used open source HEVC encoder $\times$ 265-v1.8 showed that the proposed algorithm was able to achieve an average of 14% saving in Bjøntegaard-delta-rate and an average of 20% saving in encoding time as compared the “medium” preset of $\times$ 265, while the proposed algorithm also produced an improvement of 3.3% in coding efficiency for the HEVC reference software HM-16.6. [ABSTRACT FROM AUTHOR]

Frame rate up-conversion (FRUC) is a technology for generating a higher frame rate video from one with a lower frame rate. The multi-view plus depth map (MVD) system has been recently considered a promising solution for three-dimensional video systems. Accordingly, an FRUC efficient solution for the MVD system is greatly needed. In this paper, we thus propose an FRUC approach that considers the MVD system characteristics. It divides a frame into multiple layers using a depth map, and it reconstructs occluded regions using neighboring views and depth map information. With reconstructed information, it accurately performs motion estimation when a region occlusion occurs. Experimental results demonstrated that the proposed method significantly improved the quality of interpolated frames, and it can effectively complement other conventional FRUC approaches. [ABSTRACT FROM AUTHOR]

In this paper, we propose a fast H.264-to-HEVC transcoder composed of a motion propagation algorithm and a fast mode decision framework. The motion propagation algorithm creates a motion vector candidate list at the coding tree unit (CTU) level and, thereafter, selects the best candidate at the prediction unit level. This method eliminates computational redundancy by pre-computing the prediction error of each candidate at the CTU level and reusing the information for various partition sizes. The fast mode decision framework is based on a post-order traversal of the CTU and includes several mode reduction techniques. In particular, the framework permits the early termination of the rate distortion cost computation, a highly complex task, when a mode is unpromising. Moreover, a novel method exploits the data created by the motion propagation algorithm to determine whether a coding unit must be split. This allows the pruning of unpromising sub-partitions. Compared with a cascaded pixel-domain transcoding approach, the experimental results show that the proposed solution using one reference frame is on average 8.5 times faster, for an average Bjøntegaard delta-rate (BD-Rate) of 2.63%. For a configuration with four reference frames, the average speed-up is 11.77 times and the average BD-Rate is 3.82%. [ABSTRACT FROM AUTHOR]

Vision-based lane detection is a critical component of modern automotive active safety systems. Although a number of robust and accurate lane estimation (LE) algorithms have been proposed, computationally efficient systems that can be realized on embedded platforms have been less explored and addressed. This paper presents a framework that incorporates contextual cues for LE to further enhance the performance in terms of both computational efficiency and accuracy. The proposed context-aware LE framework considers the state of the ego vehicle, its surroundings, and the system-level requirements to adapt and scale the LE process resulting in substantial computational savings. This is accomplished by synergistically fusing data from multiple sensors along with the visual data to define the context around the ego vehicle. The context is then incorporated as an input to the LE process to scale it depending on the contextual requirements. A detailed evaluation of the proposed framework on real-world driving conditions shows that the dynamic and static configuration of the lane detection process results in computation savings as high as 90%, without compromising on the accuracy of LE. [ABSTRACT FROM AUTHOR]

This paper presents a novel video-based face recognition algorithm by using a sequential sampling and updating scheme, named sequential sample consensus. The proposed algorithm aims at providing a sequential scheme that can be applied to streaming video data. Different from existing approaches, the training video sequences serve as the sample space, and the person’s identity in the testing sequence is characterized using an identity probability mass function (PMF) that is sequentially updated. For each testing frame, samples are randomly drawn from the sample space, and the numbers of samples for each identity are determined by the identity PMF. The testing frame is evaluated against the drawn samples to calculate the weights, and the sample weights are used for updating the identity PMF. Benefiting from the sampling procedure, the change in both the numbers and the weights of the samples for each individual leads to quick reaction of the algorithm. The proposed algorithm is robust against misclassification caused by pose variations, and sensitive to identity switching during recognition. The algorithm is evaluated using both public and self-made datasets, and shows better performance than other video-based face recognition approaches. [ABSTRACT FROM PUBLISHER]

This paper designs a novel underpainting based camera shooting resilient (CSR) document watermarking algorithm for dealing with the leak source tracking problem. By applying such algorithm, we can extract the authentication watermark information from the candid photographs. The watermarked underpainting contains three significant properties. 1) Inconspicuousness. The watermarked underpainting is inconspicuous and it will not easily be maliciously attacked. 2) Robustness. We propose DCT-based watermark embedding algorithm and distortion compensation based extracting algorithm, which make the watermark robust to camera shooting process. 3) Autocorrelation. We design the flip-based method to arrange the watermarked underpainting. So that a complete watermark region can be accurately located even if part of the document is recorded. Compared with previous watermarking algorithms, the proposed scheme guaranteed content independent embedding as well as the robustness to the camera shooting process. Besides, the proposed scheme satisfies the accuracy of extraction even when the captured document is incomplete. [ABSTRACT FROM AUTHOR]

Person attribute recognition aims to identify the attribute labels from the pedestrian images. Extracting contextual relation from the images and attributes, including the spatial-semantic relations, the spatial context and the semantic correlation, is beneficial to enhance the discrimination of the features for recognizing the attributes. Thus, this work proposes a sequence contextual relation learning (SCRL) method to capture these relations. It first embeds the images and attributes into sequences in two branches. Then SCRL flexibly learns the contextual relation from the sequences with the parallel attention model structure, which integrates the inter-attention and intra-attention models. The inter-attention module is utilized to extract the spatial-semantic relations, while the intra-attention is designed to gain the spatial context and the semantic correlation. Both attention modules are comprised of several parallel attention units and each unit can obtain the pairwise relations in one subspace. Therefore, they obtain the relations in multiple subspaces, which can improve the comprehensiveness of the relation learning. Additionally, for the sake of better extraction of spatial-semantic relations, this paper employs connectionist temporal classification (CTC) loss which is capable of driving the network to enforce monotonic alignment between the image and attribute. It can also accelerate the convergence of the network by the algorithm in it. Extensive experiments on five public datasets, i.e., Market-1501 attribute, Duke attribute, PETA, RAP and PA-100K datasets, demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Graph based classification methods have been widely applied in the fields of computer vision and machine learning. The quality of the graph highly affects the performance of these methods. The same object is commonly represented by different features, i.e., multi-view features, which leads to multiple graphs corresponding to different features in multi-view learning. However, what kind of graph is important for the task is unknown in advance. Moreover, existing multi-view learning methods become weak in dealing with noisy graphs when the data is corrupted by the noise. In this paper, we address this problem by observing that the noise of each graph has specific structure. Then, based on this observation we propose a robust multi-view graph recovery (RMGR) method in which the specific structure is used to clean the multiple input noisy graphs and these cleaned graphs are simultaneously aggregated into a consensus graph by adaptively assigning great weighted coefficients for important graphs. To make the consensus graph suit classification, the clustering structure is introduced to restrain the rank of Laplacian matrix of the consensus graph such that the number of its connected components is equal to that of clustering. In doing so, the graph is adaptively adjusted during optimization to more accurately partition data. The optimization problem is solved by proposed the iterative update algorithm. Extensive experiments on synthetic and several benchmark data sets show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Object recognition is widely used in vision computing for various applications. Traditional CPU and application specific integrated circuit for vision computing cannot provide high performance and enough flexibility, which limit the use of vision systems. In this paper, a hierarchical and parallel pipelined heterogeneous chip for object recognition is proposed to achieve high flexibility, high performance, and area efficiency. In addition, a reformulation of 3D position estimation is proposed. The method uses single precision to achieve the short computing time and accuracy requirement. The hardware resource is small. Application-specific components, such as connected component information extractor and information extraction accelerator, are designed for high performance. Reconfiguration processors and application-specific instruction set processor are introduced to improve flexibility. These components are connected to hierarchical parallel buses. The chip is fabricated in 180-nm CMOS technology and occupies 72.25 mm2 with 1.09M bits on-chip memory. It delivers 204 GOPS + 665M FLOPS operations. The results show that this hierarchical and parallel pipelined heterogeneous chip is suitable for embedded vision systems. [ABSTRACT FROM AUTHOR]

Video surveillance systems have become extremely important recently. It has been observed that information extracted from a single spectrum video is often insufficient in adverse conditions like low illumination, shadowing, smoke, dust, unstable background, and camouflage. Real-time video processing systems also need to be very fast where future frames are often unavailable at the time of processing the current frame. In this paper, we propose a superpixel-based causal multisensor video fusion algorithm suitable for real-time surveillance tasks. We develop new superpixel level spatial and temporal saliency models. Novel superpixel level multiple fusion rules are also designed to obtain the fused output. Comprehensive comparisons with several existing works clearly indicate the benefit of our solution. [ABSTRACT FROM AUTHOR]

Compressed sensing (CS) is the theory and practice of sub-Nyquist sampling of sparse signals of interest. Exact reconstruction may then be possible with much fewer than the Nyquist-required number of data. In this paper, we consider a multiview video system in which multiple cameras at different locations perform independent CS to simultaneously capture different views of a scene. At the decoder, we propose a disparity- and motion-compensated total variation minimization algorithm to jointly reconstruct the multiview video sequence. The experimental results show that the proposed joint reconstruction algorithm successfully exploits simultaneously intra-frame, inter-frame, and inter-view sparsity and significantly outperforms existing independent-view reconstruction, residue-view reconstruction, and motion-adaptive reconstruction algorithms. [ABSTRACT FROM AUTHOR]

Sparse modeling representative selection (SMRS) has been recently introduced for selecting the most relevant examples in data sets. SMRS exploits data self-representativeness coding in order to infer a coding matrix with block sparsity constraint. The relevance scores of samples are then derived from the estimated matrix of coefficients. Since SMRS is based on a linear model for data self-representation, it cannot always provide good relevant samples. Besides, most of its selected samples can be found in dense areas in input space. In this paper, we propose to overcome the SMRS method’s shortcomings that are related to the coding matrix estimation. We introduce two nonlinear data self-representativeness coding schemes that are based on Hilbert space and column generation. Experimental evaluation is carried out on summarizing a video movie and on summarizing training image data sets used for classification tasks. These experiments demonstrated that the proposed nonlinear methods can outperform state-of-the art selection methods. [ABSTRACT FROM AUTHOR]

A residual networks family with hundreds or even thousands of layers dominates major image recognition tasks, but building a network by simply stacking residual blocks inevitably limits its optimization ability. This paper proposes a novel residual network architecture, residual networks of residual networks (RoR), to dig the optimization ability of residual networks. RoR substitutes optimizing residual mapping of residual mapping for optimizing original residual mapping. In particular, RoR adds levelwise shortcut connections upon original residual networks to promote the learning capability of residual networks. More importantly, RoR can be applied to various kinds of residual networks (ResNets, Pre-ResNets, and WRN) and significantly boost their performance. Our experiments demonstrate the effectiveness and versatility of RoR, where it achieves the best performance in all residual-network-like structures. Our RoR-3-WRN58-4 + SD models achieve new state-of-the-art results on CIFAR-10, CIFAR-100, and SVHN, with the test errors of 3.77%, 19.73%, and 1.59%, respectively. RoR-3 models also achieve state-of-the-art results compared with ResNets on the ImageNet data set. [ABSTRACT FROM AUTHOR]

Low-dose computed tomography (LDCT) images are often highly degraded by amplified mottle noise and streak artifacts. Maintaining image quality under low-dose scan protocols is a well-known challenge. Recently, sparse representation-based techniques have been shown to be efficient in improving such CT images. In this paper, we propose a 3D feature constrained reconstruction (3D-FCR) algorithm for LDCT image reconstruction. The feature information used in the 3D-FCR algorithm relies on a 3D feature dictionary constructed from available high quality standard-dose CT sample. The CT voxels and the sparse coefficients are sequentially updated using an alternating minimization scheme. The performance of the 3D-FCR algorithm was assessed through experiments conducted on phantom simulation data and clinical data. A comparison with previously reported solutions was also performed. Qualitative and quantitative results show that the proposed method can lead to a promising improvement of LDCT image quality. [ABSTRACT FROM PUBLISHER]

Marketing over online social networks (OSNs) has become an essential tool for spreading product information in a “word of mouth” way. In particular, campaigns normally adopt a pragmatic approach of seeding videos with a selected list of influential users, hoping to create a viral distribution to reach as many users as possible. In this paper, we propose a multitopic-aware influence maximization framework to identify a fixed number of influential users and assign video clips of specific topics to them, with an ultimate objective to maximize the number of message deliveries, defined as expected posting number (EPN). We first prove the submodularity of the EPN function, resulting in a general greedy algorithm with a performance bound of $1-1/e$ . We further develop two faster algorithms to accelerate the computing speed for large-scale social networks. The first algorithm leverages two estimation methods to compute the upper bound for marginal EPN without a loss of accuracy. The second algorithm generates an approximation solution based on the upper bound and lower bound estimation, with a performance bound of $\epsilon (1-1/e)$ . We have implemented a prototype system based on a private data center at the Nanyang Technological University campus in Singapore to enable video clip extraction and sharing among social users. Furthermore, we conduct experiments on four real large-scale social networks (with different scales and structures) and the results show that the proposed methods are much faster than previous algorithms but with high accuracy. [ABSTRACT FROM PUBLISHER]

Object tracking in a video sequence is usually achieved by tracking the bounding box over the object or the object’s boundary, each of which has somewhat different applications. In this paper, we present a new open-contour tracking algorithm based on a Bayesian framework in which the contour is a part of the object’s boundary. We first propose a new state-space model for the representation of contours, which can handle the rigid and nonrigid motions of contours independently. This model enables us to focus on the nonrigid motions during the training, and the model works for challenging rigid motion scenarios. In addition, for the robust tracking of contours, we propose a measurement function that considers the contrast on object boundaries, target appearance, and temporal coherence. We applied the proposed method to two kinds of open-contours targets, and the experimental results show that the proposed method achieves superior performance to the conventional contour tracking methods. The proposed method is also compared with recent bounding box tracking methods for the object tracking purposes, and the comparison shows that the proposed method works robustly to fast motions and yields a more accurate estimate of an object’s location than the conventional bounding box tracking methods. [ABSTRACT FROM AUTHOR]

As a specific data hiding technique, reversible data hiding (RDH) has recently received extensive attention. By this technique, both the embedded data and the original cover image can be exactly extracted from the marked image. In our previous work, a general expansion-shifting model for RDH is proposed by introducing the so-called reversible embedding function (REF). With REF, RDH can be designed and the corresponding rate-distortion formulations can be established, providing an approach to optimize the reversible embedding performance. In this paper, by extending our previous work, optimal REF for one-dimensional histogram is investigated, and all optimal REF are derived in this case when the maximum modification to the cover pixel is limited as a small value. Moreover, based on the derived optimal REF for one-dimensional histogram and multiple histograms modification, a practical RDH scheme is presented and it is experimental verified better than some state-of-the-art algorithms in terms of capacity-distortion performance. [ABSTRACT FROM AUTHOR]

The article presents an introduction to the second issue on image-and video-based biometrics, published in February 1, 2004 issue of "IEEE Transactions in Circuits and Systems for Video Technology." In this issue, authors present a set of papers addressing relatively newer technologies including gait, human body dynamics, palmprint, paim-dorsa vein-patterns and multimodal algorithms. In the algorithm, the static and dynamic cues obtained from walking video are fused at the decision level using different combination rules and are shown to improve the performance of human identification and verification.

Joint sparse representation (JSR) model has recently emerged as a powerful technique with wide variety of applications. In this paper, the JSR model is extended to error concealment (EC) application, being effective to recover the original image from its corrupted version. This model is based on jointly learning a dictionary pair and two mapping matrices that are trained offline from external training images. Given the trained dictionaries and mappings, the restoration is done by transferring the recovery problem into the sparse representation domain with respect to the trained dictionaries, which is further transformed into a common space using the respective mapping matrices. Then, the reconstructed image is obtained by back projection into the spatial domain. In order to improve the accuracy and stability of the proposed JSR-based EC algorithm and avoid unexpected artifacts, the local and non-local priors are seamlessly integrated into the JSR model. The non-local prior is based on the self-similarity within natural images and helps to find an accurate sparse representation by taking a weighted average of similar areas throughout the image. The local prior is based on learning the local structural regularity of the natural images and helps to regularize the sparse representation, exploiting the strong correlation in the small local areas within the image. Compared with the state-of-the-art EC algorithms, the results show that the proposed method has better reconstruction performance in terms of objective and subjective evaluations. [ABSTRACT FROM AUTHOR]

It is recognized that existing multi-scale exposure fusion algorithms can be improved using edge-preserving smoothing techniques. However, the complexity of edge-preserving smoothing-based multi-scale exposure fusion is an issue for mobile devices. In this paper, a simpler multi-scale exposure fusion algorithm is designed in YUV color space. The proposed algorithm can preserve details in the brightest and darkest regions of a high dynamic range (HDR) scene and the edge-preserving smoothing-based multi-scale exposure fusion algorithm while avoiding color distortion from appearing in the fused image. The complexity of the proposed algorithm is about half of the edge-preserving smoothing-based multi-scale exposure fusion algorithm. The proposed algorithm is thus friendlier to the smartphones than the edge-preserving smoothing-based multi-scale exposure fusion algorithm. In addition, a simple detail-enhancement component is proposed to enhance fine details of fused images. The experimental results show that the proposed component can be adopted to produce an enhanced image with visibly enhanced fine details and a higher MEF-SSIM value. This is impossible for existing detail enhancement components. Clearly, the component is attractive for PC-based applications. [ABSTRACT FROM AUTHOR]

Dictionary learning has emerged as a powerful tool for a range of image processing applications and a proper dictionary always plays a key issue to the final achievable performance. In this paper, a class-oriented discriminative dictionary learning (CODDL) method is presented for image classification applications. It takes a comprehensive consideration of multiple optimization objectives, emphasizing class discrimination of both dictionary atoms and representation coefficients. The atoms of the learned dictionary should be grouped into class level sub-dictionaries. Meanwhile, the sparse representation coefficients of an input sample should be concentrated on the sub-dictionary of the class it belongs to. Then, based on the learned class-oriented discriminative dictionary, the structured representation coefficients can thus be used for image classification with a simple and efficient classification scheme. The superior performance of the proposed algorithm is demonstrated through extensive experiments. [ABSTRACT FROM AUTHOR]

This paper proposes and evaluates a novel algorithm for streaming video over HTTP. The problem is formulated as a non-convex optimization problem which is constrained by the predicted available bandwidth, chunk deadlines, available video rates, and buffer occupancy. The objective is to optimize a QoE metric that maintains a tradeoff between maximizing the playback rate of every chunk and ensuring fairness among different chunks for the minimum re-buffering time. We propose FastScan, a low complexity algorithm that solves the problem. The online adaptations for dynamic bandwidth environments are proposed with imperfect available bandwidth prediction. The results of experiments driven by variable bit rate (VBR) encoded video, video platform system (dash.js), and cellular bandwidth traces of a public dataset reveal the robustness of the online version of FastScan algorithm and demonstrate its significant performance improvement as compared to the considered state-of-the-art video streaming algorithms. For example, on an experiment conducted over 100 real cellular available bandwidth traces of a public dataset that spans different available bandwidth regimes, our proposed algorithm (FastScan) achieves the minimum re-buffering (stall) time and the maximum average playback rate in every single trace as compared to Bola, Festive, BBA, RB, FastMPC, and Pensieve algorithms. [ABSTRACT FROM AUTHOR]

Wavelet transform using Haar filter is a fast process for decomposing an image into low and high frequency sub-bands, which is an important step in multi-focus image fusion. This transformation alone is usually not sufficient to produce satisfying fused images, and therefore additional tools like focused region decision map and feature extraction are needed to enhance the fusion quality. In this paper, we propose a multi-focus image fusion algorithm that is suitable for efficient hardware implementation. The algorithm speed is significantly improved by utilizing Haar wavelet and computationally simple fusion rules. To further improve the circuit power and delay, we limit the logic operations to additions, subtractions, and multiplications only. Experiments on different benchmarks demonstrate that our proposed algorithm can reduce the fusion time by up to 99.97% compared with the state-of-the-art competing methods without compromising the fusion quality. In addition, the hardware implementation of the proposed algorithm reduces the circuit area and delay by 79.5% and 92.5%, respectively, when compared to the most competitive algorithm in this paper while maintaining similar fusion quality. [ABSTRACT FROM AUTHOR]

In this paper, we propose a lossy image compression algorithm called microshift. We employ an algorithm-hardware co-design methodology, yielding a hardware-friendly compression approach with low power consumption. In our method, the image is first micro-shifted, and then the sub-quantized values are further compressed. Two methods, FAST and MRF models, are proposed to recover the bitdepth by exploiting the spatial correlation of natural images. Both methods can decompress images progressively. On an average, our compression algorithm can compress images to 1.25-bits per pixel with a resulting quality that outperforms the state-of-the-art on-chip compression algorithms in both peak signal-to-noise ratio and structual similarity. Then, we propose a hardware architecture and implement the algorithm on an FPGA. The results on the ASIC design further validate the low-hardware complexity and high-power efficiency, showing that our method is promising, particularly for low-power wireless vision sensor networks. [ABSTRACT FROM AUTHOR]