Search

Low-light image enhancement algorithms can improve the visual quality of low-light images and support the extraction of valuable information for some computer vision techniques. However, existing techniques inevitably introduce color and lightness distortions when enhancing the images. To lower the distortions, we propose a novel enhancement framework using the response characteristics of cameras. First, we discuss how to determine a reasonable camera response model and its parameters. Then, we use the illumination estimation techniques to estimate the exposure ratio for each pixel. Finally, the selected camera response model is used to adjust each pixel to the desired exposure according to the estimated exposure ratio map. Experiments show that our method can obtain enhancement results with fewer color and lightness distortions compared with the several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Deep learning methods, especially CNNs, have achieved many promising results in a wide range of computer vision applications. However, few studies focused on designing suitable deep learning methods for parking space status inference. As we have known, it is challenging to detect parking spaces in an outdoor environment due to dynamic lighting variations, weather changes, and perspective distortion. By off-the-shelf CNNs, lighting variations might be handled well. However, to realize a practical and robust inference system, we also need to address troublesome problems, such as parking displacements, non-unified car sizes, inter-object occlusion, and perspective distortion. These problems may become even challenging if also considering the difference of space sizes. To overcome the problems, we proposed a custom–tailored deep convolutional and contrastive network with three contributions. First, we introduced a Siamese architecture to learn the contrastive and robust feature descriptor. This helps to reduce the effects owing to the variety of inter-object occlusion. Second, we integrated a convolutional Spatial Transformer Network (STN) to adaptively transform a 3-space input patch according to vehicle sizes and parking displacement. STN also helps to overcome the perspective distortion problem. Third, a multi-task loss function was designed to train the network by simultaneously considering the accuracy of inferring the status of the target space and the semantic smoothness of high-level features. Thereby, the errors caused by inter-object occlusion could be alleviated. To verify the proposed network, we visualized the learned features and analyzed their functionality. Experiments and evaluations have shown the robustness of our system in parking status inference. The real-time system currently running in public parking lots also demonstrates the effectiveness of the proposed deep network. [ABSTRACT FROM AUTHOR]

Advanced wireless imaging sensors and cloud data storage contribute to video surveillance by enabling the generation of large amounts of video footage every second. Consequently, surveillance videos have become one of the largest sources of unstructured data. Because multi-scenario surveillance videos are often continuously produced, using these videos to detect moving objects is challenging for the conventional moving object detection methods. This paper presents a novel model that harnesses both sparsity and low-rankness with contextual regularization to detect moving objects in multi-scenario surveillance data. In the proposed model, we consider moving objects as a contiguous outlier detection problem through the use of low-rank constraint with contextual regularization, and we construct dedicated backgrounds for multiple scenarios using dictionary learning-based sparse representation, which ensures that our model can be effectively applied to multi-scenario videos. Quantitative and qualitative assessments indicate that the proposed model outperforms existing methods and achieves substantially more robust performance than the other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Multiview applications endow final users with the possibility to freely navigate within 3D scenes with minimum-delay. A real feeling of scene navigation is enabled by transmitting multiple high-quality camera views, which can be used to synthesize additional virtual views to offer a smooth navigation. However, when network resources are limited, not all camera views can be sent at high quality. It is therefore important, yet challenging, to find the right tradeoff between coding artifacts (reducing the quality of camera views) and virtual synthesis artifacts (reducing the number of camera views sent to users). To this aim, we propose an optimal transmission strategy for interactive multiview HTTP adaptive streaming. We propose a problem formulation to select the optimal set of camera views that the client requests for downloading, such that the navigation quality experienced by the user is optimized while the bandwidth constraints are satisfied. We show that our optimization problem is NP-hard, and we therefore develop an optimal solution based on the dynamic programming algorithm with polynomial time complexity. To further simplify the deployment, we present a suboptimal greedy algorithm with effective performance and lower complexity. The proposed controller is evaluated in theoretical and realistic settings characterized by realistic network statistics estimation, buffer management, and server-side representation optimization. Simulation results show significant improvement in terms of navigation quality compared with alternative baseline multiview adaptation logic solutions. [ABSTRACT FROM AUTHOR]

2-D linear discriminant analysis (2DLDA) has been widely used in pattern recognition and image classification. 2DLDA selects discriminative features from the up and left corner of images. However, 2DLDA uses the Frobenius norm (F-norm), which is sensitive to noise or outliers in data, as a metric. In this paper, we propose a novel framework, called horizontal and vertical nuclear norm-based 2DLDA (HVNN-2DLDA) for image representation. In the proposed framework, HVNN-2DLDA methods (i.e., HNN-2DLDA and VNN-2DLDA) are proposed, and both use the nuclear norm as a criterion. The nuclear norm can provide more structure and global information for the reconstruction of noisy images. HNN-2DLDA and VNN-2DLDA represent images in the row and column directions, respectively. In addition, by combining the row and column directions, we propose a bilateral nuclear norm-based 2DLDA method called BNN-2DLDA. The advantage of BNN-2DLDA over HNN-2DLDA and VNN-2DLDA is that an image sample can be represented by both the row and the column directions instead of only the row or column direction. HVNN-2DLDA learns a set of local optimal projection vectors by maximizing the ratio of the nuclear norm of the between-class scatter matrix and the nuclear norm of the within-class scatter matrix. To verify the robustness and recognition performance in image classification of HVNN-2DLDA, six public image databases are used for experiments. The experimental results demonstrate the effectiveness and the feasibility of the proposed framework. [ABSTRACT FROM AUTHOR]

Depth image-based rendering (DIBR) techniques have recently drawn more attention in various 3D applications. In this paper, a real-time high-quality DIBR system that consists of disparity estimation and view synthesis is proposed. For disparity estimation, a local approach that focuses on depth discontinuities and disparity smoothness is presented to improve the disparity accuracy. For view synthesis, a method that contains view interpolation and extrapolation is proposed to render high-quality virtual views. Moreover, the system is designed with an optimized parallelism scheme to achieve a high throughput, and can be scaled up easily. It is implemented on an Altera Stratix IV FPGA at a processing speed of 45 frames per second for 1080p resolution. Evaluated on selected image sets of the Middlebury benchmark, the average error rate of the disparity maps is 6.02%; the average peak signal to noise ratio and structural similarity values of the virtual views are 30.07 dB and 0.9303, respectively. The experimental results indicate that the proposed DIBR system has the top-performing processing speed and its accuracy performance is among the best of state-of-the-art hardware implementations. [ABSTRACT FROM AUTHOR]

Despite recent advances made in salient object detection using graph theory, the approach still suffers from accuracy problems when the image is characterized by a complex structure, either in the foreground or background, causing erroneous saliency segmentation. This fundamental challenge is mainly attributed to the fact that most existing graph-based methods take only the adjacently spatial consistency among graph nodes into consideration. In this paper, we tackle this issue from a coarse-to-fine perspective and propose a two-stage-graphs approach for salient object detection, in which two graphs having the same nodes but different edges are employed. Specifically, a weighted joint robust sparse representation model, rather than the commonly used manifold ranking model, helps to compute the saliency value of each node in the first-stage graph, thereby providing a saliency map at the coarse level. In the second-stage graph, along with the adjacently spatial consistency, a new regionally spatial consistency among graph nodes is considered in order to refine the coarse saliency map, assuring uniform saliency assignment even in complex scenes. Particularly, the second stage is generic enough to be integrated in existing salient object detectors, enabling improved performance. Experimental results on benchmark data sets validate the effectiveness and superiority of the proposed scheme over related state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, we present a comparative evaluation of various multi-person tracking-by-detection approaches on public data sets. This paper investigates five popular trackers coupled with six relevant visual people detectors evaluated on seven public data sets. The evaluation emphasizes on exhibited performance variation depending on tracker-detector choices. Our experimental results show that the overall performance depends on how challenging the data set is, the performance of the detector on the specific data set, and the tracker-detector combination. Some trackers are more sensitive to the choice of a detector and some detectors to the choice of a tracker than others. Based on our results, two of the trackers demonstrate the best performances consistently across different data sets, whereas the best performing detectors vary per data set. This underscores the need for careful application context specific evaluation when choosing a detector. [ABSTRACT FROM AUTHOR]

Dynamic scene classification has been extensively studied in computer vision due to its widespread applications. The key to dynamic scene classification lies in jointly characterizing spatial appearance and temporal dynamics to achieve informative representation, which remains an outstanding task in the literature. In this paper, we propose a unified framework to extract spatial and temporal features for dynamic scene representation. More specifically, we deploy two variants of deep convolutional neural networks to encode spatial appearance and short-term dynamics into short-term deep features (STDF). Based on STDF, we propose using the autoregressive moving average model to extract long-term frequency features (LTFF). By combining STDF and LTFF, we establish the long–short-term feature (LSTF) representations of dynamic scenes. The LSTF characterizes both spatial and temporal patterns of dynamic scenes for comprehensive and information representation that enables more accurate classification. Extensive experiments on three-dynamic scene classification benchmarks have shown that the proposed LSTF achieves high performance and substantially surpasses the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Common non-focused areas are often present in multi-focus images due to the limitation of the number of focused images. This factor severely degrades the fusion quality of multi-focus images. To address this problem, we propose a novel end-to-end multi-focus image fusion with a natural enhancement method based on deep convolutional neural network (CNN). Several end-to-end CNN architectures that are specifically adapted to this task are first designed and researched. On the basis of the observation that low-level feature extraction can capture low-frequency content, whereas high-level feature extraction effectively captures high-frequency details, we further combine multi-level outputs such that the most visually distinctive features can be extracted, fused, and enhanced. In addition, the multi-level outputs are simultaneously supervised during training to boost the performance of image fusion and enhancement. Extensive experiments show that the proposed method can deliver superior fusion and enhancement performance than the state-of-the-art methods in the presence of multi-focus images with common non-focused areas, anisotropic blur, and misregistration. [ABSTRACT FROM AUTHOR]

Despite the recent success of 3D action recognition using depth sensor, most existing works target how to improve the action recognition performance, rather than understanding how different types of actions are performed. In this paper, we propose to discover discriminative spatio-temporal patterns for 3D action recognition. Discovering these patterns can not only help to improve the action recognition performance but also help us to understand and differentiate between the action category. Our proposed method takes the spatio-temporal structure of 3D action into consideration and can discover essential spatio-temporal patterns that play key roles in action recognition. Instead of relying on an end-to-end network to learn the 3D action representation and perform classification, we simply present each 3D action as a series of temporal stages composed by 3D poses. Then, we rely on nearest neighbor matching and bilinear classifiers to simultaneously identify both critical temporal stages and spatial joints for each action class. Despite using raw action representation and a linear classifier, experiments on five benchmark data sets show that the proposed spatio-temporal naïve Bayes mutual information maximization can achieve a competitive performance compared with the state-of-the-art methods that use sophisticated end-to-end learning, and has the advantage of finding discriminative spatio-temporal action patterns. [ABSTRACT FROM AUTHOR]

Achieving good text detection and recognition results for multi-script-oriented images is a challenging task. First, we explore bit plane slicing in order to utilize the advantage of the most significant bit information to identify text components. A new iterative nearest neighbor symmetry is then proposed based on shapes of convex and concave deficiencies of text components in bit planes to identify candidate planes. Further, we introduce a new concept called mutual nearest neighbor pair components based on gradient direction to identify representative pairs of texts in each candidate bit plane. The representative pairs are used to restore words with the help of edge image of the input one, which results in text detection results (words). Second, we propose a new idea by fixing window for character components of arbitrary oriented words based on angular relationship between sub-bands and a fused band. For each window, we extract features in contourlet wavelet domain to detect characters with the help of an SVM classifier. Further, we propose to explore HMM for recognizing characters and words of any orientation using the same feature vector. The proposed method is evaluated on standard databases such as ICDAR, YVT video, ICDAR, SVT, MSRA scene data, ICDAR born digital data, and multi-lingual data to show its superiority to the state of the art methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel deep learning-based visual comfort assessment (VCA) for stereoscopic images. To assess the overall degree of visual discomfort in stereoscopic viewing, we devise a binocular fusion deep network (BFN) learning binocular characteristics between stereoscopic images. The proposed BFN learns the latent binocular feature representations for the visual comfort score prediction. In the BFN, the binocular feature is encoded by fusing the spatial features extracted from left and right views. Finally, the visual comfort score is predicted by projecting the binocular feature onto the subjective score space. In addition, we devise a disparity regularization network (DRN) for improving the prediction results. The proposed DRN takes the binocular feature from the BFN and estimates disparity maps from the feature in order to embed disparity relations between left and right views into the deep network. The proposed deep network with BFN and DRN is end-to-end trained in a unified framework in which the DRN acts as disparity regularization. We evaluated the prediction performance of the proposed deep network for VCA by the comparison of existing objective VCA metrics. Further, we demonstrated that the proposed BFN showed various factors causing visual discomfort by using network visualization. [ABSTRACT FROM AUTHOR]

Detection of relative changes in circulating blood volume is important to guide resuscitation and manage a variety of medical conditions, including sepsis, trauma, dialysis, and congestive heart failure. Recent studies have shown that estimates of circulating blood volume can be obtained from the cross-sectional area of the internal jugular vein (IJV) from ultrasound images. However, accurate segmentation and tracking of the IJV in ultrasound imaging is a challenging task and is significantly influenced by a number of parameters, such as the image quality, shape, and temporal variation. In this paper, we propose a novel adaptive polar active contour (Ad-PAC) algorithm for the segmentation and tracking of the IJV in ultrasound videos. In the proposed algorithm, the parameters of the Ad-PAC algorithm are adapted based on the results of segmentation in previous frames. The Ad-PAC algorithm is applied to 65 ultrasound videos captured from 13 healthy subjects, with each video containing 450 frames. The results show that spatial and temporal adaptation of the energy function significantly improves segmentation performance when compared with the current state-of-the-art active contour algorithms. [ABSTRACT FROM AUTHOR]

We present a high-quality temporal frame interpolation (TFI) method that employs piecewise-smooth motion and handles disoccluded regions using the observation that motion discontinuities travel with the foreground object. We derive a “motion discontinuity” likelihood map from the divergence of a motion field between the input frames. Motion which is modeled at the reference frame is mapped to the target frame using a cellular-affine mapping strategy—a process during which regions of disocclusion are readily observed. This information is then used to guide the occlusion-aware, bidirectional FI process. Furthermore, we propose two computationally inexpensive texture optimizations that selectively improve the quality of the interpolated frames in regions around moving objects. The scheme produces very high-quality interpolated frames and outperforms current high-quality state-of-the-art TFI schemes by 2–2.5 dB; the method works with a very low-complexity motion estimation scheme and runs orders of magnitudes faster than its competitors. [ABSTRACT FROM AUTHOR]

We investigate video classification via a two-stream convolutional neural network (CNN) design that directly ingests information extracted from compressed video bitstreams. Our approach begins with the observation that all modern video codecs divide the input frames into macroblocks (MBs). We demonstrate that selective access to MB motion vector (MV) information within compressed video bitstreams can also provide for selective, motion-adaptive, MB pixel decoding (a.k.a., MB texture decoding). This in turn allows for the derivation of spatio-temporal video activity regions at extremely high speed in comparison to conventional full-frame decoding followed by optical flow estimation. In order to evaluate the accuracy of a video classification framework based on such activity data, we independently train two CNN architectures on MB texture and MV correspondences and then fuse their scores to derive the final classification of each test video. Evaluation on two standard data sets shows that the proposed approach is competitive with the best two-stream video classification approaches found in the literature. At the same time: 1) a CPU-based realization of our MV extraction is over 977 times faster than GPU-based optical flow methods; 2) selective decoding is up to 12 times faster than full-frame decoding; and 3) our proposed spatial and temporal CNNs perform inference at 5 to 49 times lower cloud computing cost than the fastest methods from the literature. [ABSTRACT FROM AUTHOR]

The active contour model is a widely used technique for automatic object contour extraction. Existing methods based on this model can perform with high accuracy, even in the case of complex contours, but challenging issues remain, like the need for precise contour initialization for high curvature boundary segments or the handling of cluttered backgrounds. To deal with such issues, this paper presents a salient object extraction method, the first step of which is the introduction of an improved edge map that incorporates edge direction as a feature. The direction information in the small neighborhoods of image feature points is extracted, and the images’ prominent orientations are defined for direction-selective edge extraction. Using such improved edge information, we provide a highly accurate shape contour representation, which we also combine with texture features. The principle of the paper is to interpret an object as the fusion of its components: its extracted contour and its inner texture. Our goal in fusing textural and structural information is twofold: it is applied for automatic contour initialization, and it is also used to establish an improved external force field. This fusion then produces highly accurate salient object extractions. We performed extensive evaluations, which confirm that the presented object extraction method outperforms parametric active contour models and achieves higher efficiency than the majority of the evaluated automatic saliency methods. [ABSTRACT FROM AUTHOR]

High-order graph matching is a feature-matching algorithm that uses geometric information among features. This algorithm is more robust to repetitive patterns or unclear areas than first-order matching that uses only feature descriptors. However, the processing speed of high-order matching is very slow because of its high computational complexity. To accelerate its speed, this paper proposes a new parallelization algorithm of high-order matching for GPU execution. The obstacle for parallelization is the write collision caused by multiple threads that must simultaneously update the data at the same memory location. In high-order matching, multiple formulations of the objective function can generate the same solution. By taking advantage of this property, the proposed algorithm replaces the operation causing write collision with another operation eliminating the collision while generating the same solution. The proposed algorithm is tested with GTX 960 and takes 31.3 ms, which is 68 times faster than the execution time with a CPU and approximately three times faster than that with a straightforward parallelization for the same GPU. [ABSTRACT FROM AUTHOR]

Compared with conventional color images, light field images (LFIs) contain richer scene information, which allows a wide range of interesting applications. However, such additional information is obtained at the cost of generating substantially more data, which poses challenges to both data storage and transmission. In this paper, we propose a new hybrid framework for effective compression of LFIs. The proposed framework takes the particular characteristics of LFIs into account so that the inter- and intra-view correlations of LFIs can be more efficiently exploited to produce better compression performance. Specifically, the proposed scheme partitions sub-aperture images (SAIs) of an LFI into two groups, namely, key SAIs and non-key SAIs. Bi-level view compensation is proposed to exploit the inter-view correlation: first, based on the group of selected key SAIs, learning-based angular super-resolution is performed to compensate non-key SAIs in pixel-wise, during which heterogeneous inter-view correlation between the non-key SAIs is efficiently removed; second, the two groups of SAIs are respectively reorganized as pseudo-sequences, and block-wise motion compensation is carried out with a standard video encoder, during which the homogeneous inter-view correlation is subsequently exploited. The video encoder also helps to remove the intra-view correlation of the SAIs and finally generates the encoded bitstream. Moreover, the bits allocated to each group are optimally determined via model-based rate distortion optimization. Extensive experimental evaluations and comparisons demonstrate the advantage of the proposed framework over existing methods in terms of rate-distortion performance. [ABSTRACT FROM AUTHOR]

This paper aims at learning discriminative part detectors with only image-level labels. To this end, we need to develop effective technologies for both pattern mining and detection learning. Different from previous methods, which train part detectors in one step, we divide the detector learning process into two stages and formulate it as a weak to strong learning framework. In particular, we first learn exemplar detectors from the unaligned patterns and perform a detector-based spectral clustering to produce weak detectors that are only responsible for a few discriminative patterns. In this way, the weak detectors are able to offer right initial patterns for strong detector learning. Second, we learn strong detectors with patterns discovered from the weak detectors, which we formulate as a confidence-loss sparse multiple instance learning (cls-MIL) task. The cls-MIL considers the diversity of positive samples while avoiding drifting away from the well localized ones by assigning a confidence value to each positive sample. The responses of the learned detectors produce an effective mid-level image representation for both image classification and object localization. Experiments conducted on benchmark data sets well demonstrate the superiority of our method over existing approaches. [ABSTRACT FROM AUTHOR]

This paper proposes a novel framework of reversible data hiding in encrypted JPEG bitstream. We first provide a JPEG encryption algorithm to encipher a JPEG image to a smaller size and keep the format compliant to JPEG decoders. After an image owner uploads the encrypted JPEG bitstreams to cloud storage, the server embeds additional messages into the ciphertext to construct a marked encrypted JPEG bitstream. During data hiding, we propose a combined embedding algorithm including two stages, the Huffman code mapping and the ordered histogram shifting. The embedding procedure is reversible. When an authorized user requires a downloading operation, the server extracts additional messages from the marked encrypted JPEG bitstream and recovers the original encrypted bitstream losslessly. After downloading, the user obtains the original JPEG bitstream by a direct decryption. The proposed framework outperforms previous works on reversible data hiding in encrypted images. First, since the tasks of data embedding/extraction and bitstream recovery are all accomplished by the server, the image owner and the authorized user are required to implement no extra operations except JPEG encryption or decryption. Second, the embedding payload is larger than state-of-the-art works. [ABSTRACT FROM AUTHOR]

Steganography detection based on Rich Model features is a hot research direction in steganalysis. However, rich model features usually result a large computation cost. To reduce the dimension of steganalysis features and improve the efficiency of steganalysis algorithm, differing from previous works that normally proposed new feature extraction algorithm, this paper proposes a general steganalysis feature selection method based on decision rough set $\alpha$ -positive region reduction. First, it is pointed out that decision rough set $\alpha$ -positive region reduction is suitable for steganalysis feature selection. Second, a quantization method of attribute separability is proposed to measure the separability of steganalysis feature components. Third, steganalysis feature components selection algorithm based on decision rough set $\alpha$ -positive region reduction is given; thus, stego images can be detected by the selected feature. The proposed method can significantly reduce the feature dimensions and maintain detection accuracy. Based on the BOSSbase-1.01 image database of 10 000 images, a series of feature selection experiments are carried on two kinds of typical rich model features (35263-D J+SRM feature and 17000-D GFR feature). The results show that even though these two kinds of features are reduced to approximately 8000-D, the detection performance of steganalysis algorithms based on the selected features are also maintained with that of original features, which will remarkably improve the efficiency of feature extraction and stego image detection. [ABSTRACT FROM AUTHOR]

Linear discriminant analysis (LDA) is a very popular supervised feature extraction method and has been extended to different variants. However, classical LDA has the following problems: 1) The obtained discriminant projection does not have good interpretability for features; 2) LDA is sensitive to noise; and 3) LDA is sensitive to the selection of number of projection directions. In this paper, a novel feature extraction method called robust sparse linear discriminant analysis (RSLDA) is proposed to solve the above problems. Specifically, RSLDA adaptively selects the most discriminative features for discriminant analysis by introducing the $l_{2,1}$ norm. An orthogonal matrix and a sparse matrix are also simultaneously introduced to guarantee that the extracted features can hold the main energy of the original data and enhance the robustness to noise, and thus RSLDA has the potential to perform better than other discriminant methods. Extensive experiments on six databases demonstrate that the proposed method achieves the competitive performance compared with other state-of-the-art feature extraction methods. Moreover, the proposed method is robust to the noisy data. [ABSTRACT FROM AUTHOR]

A residue number system (RNS) has an inherent parallel structure that can be utilized for improving computer hardware systems. An RNS represents large integer numbers as a smaller integer set, or residues of a modulo set, without carry propagation between them. Hence mathematical operations, such as addition or subtraction, can be performed on the residues independently. This paper proposes an RNS implementation of motion estimation for the latest video coding standard known as high-efficiency video coding (HEVC) or H.265. Since motion estimation is the most computationally intensive task in video coding, several simplified algorithms are proposed for mitigating the problem, but the majority of them result in a worsening peak signal-to-noise ratio (PSNR) or bit-rate performance, or sometimes both. This paper also proposes a modified algorithm based on a test-zone (TZ) search algorithm, a widely used fast-search algorithm with good rate-distortion performance, suitable for hardware implementation for encoding ultra-high-definition videos in real time. The results show that worst-case PSNR degradation and bit-rate increases compared with the TZ search in the HEVC reference software implementation are negligible, and the hardware gate count is less than for many other designs in the literature. [ABSTRACT FROM AUTHOR]

Although encouraging results have been obtained in human pose estimation in recent years, the performance may degrade dramatically when the image quality differs between training and testing data sets. This paper addresses problems in cross-image-quality human pose estimation. To achieve this, we follow an unsupervised domain adaptation approach, in which labels in the target domain are unavailable. Unlike existing unsupervised domain adaptation methods that find label information from unlabeled data, the target pose information (label) is instead generated by synthesizing body parts with similar image-quality of the target domain. A translative dictionary is learned to associate the source and target domains, and a cross-quality adaptation model is developed to refine the source pose estimator using the synthesized target body parts. We perform cross-quality experiments on three data sets with different image quality by using two state-of-the-art pose estimators, and compare the proposed method with five unsupervised domain adaptation methods. Our experimental results show that the proposed method outperforms not only the source pose estimators, but also other unsupervised domain adaptation methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose an adaptive perceptual quantization method to convert the representation of high dynamic range (HDR) content from the floating point data type to integer, which is compatible with the current image/video coding and display systems. The proposed method considers the luminance distribution of the HDR content, as well as the detectable contrast threshold of the human visual system, in order to preserve more contrast information than the perceptual quantizer (PQ) in integer representation. Aiming to demonstrate the effectiveness of this quantizer for HDR video compression, we implemented it in a mapping function on the top of the HDR video coding system based on high efficiency video coding standard. Moreover, a comparison function is also introduced to decrease the additional bit-rate of side information, generated by the mapping function. Objective quality measurements and subjective tests have been conducted in order to evaluate the quality of the reconstructed HDR videos. Subjective test results have shown that the proposed method can improve, in a significant manner, the perceived quality of some reconstructed HDR videos. In the objective assessment, the proposed method achieves improvements over PQ in terms of the average bit-rate gain for metrics used in the measurement. [ABSTRACT FROM AUTHOR]

In this paper, we propose an ordinal deep learning approach for facial age estimation. Unlike conventional hand-crafted feature-based methods that require prior and expert knowledge, we propose an ordinal deep feature learning (ODFL) method to learn feature descriptors for face representation directly from raw pixels. Motivated by the fact that age labels are chronologically correlated and age estimation is an ordinal learning problem, our proposed ODFL enforces two criteria on the descriptors, which are learned at the top of the deep networks: 1) the topology-preserving ordinal relation is employed to exploit the order information in the learned feature space and 2) the age-difference cost information is leveraged to dynamically measure face pairs with different age value gaps. However, both the procedures of feature extraction and age estimation are learned independently in ODFL, which may lead to a sub-optimal problem. To address this, we further propose an end-to-end ordinal deep learning (ODL) framework, where the complementary information of both the procedures is exploited to reinforce our model. Extensive experimental results on five face aging datasets show that both our ODFL and ODL achieve superior performance in comparisons with most state-of-the-art methods. [ABSTRACT FROM AUTHOR]

High dynamic range (HDR) image, which has a powerful capacity to represent the wide dynamic range of real-world scenes, has been receiving attention from both academic and industrial communities. Although HDR imaging devices have become prevalent, the display devices for HDR images are still limited. To facilitate the visualization of HDR images in standard low dynamic range displays, many different tone mapping operators (TMOs) have been developed. To create a fair comparison of different TMOs, this paper proposes a BLInd QUality Evaluator to blindly predict the quality of Tone-Mapped Images (BLIQUE-TMI) without accessing the corresponding HDR versions. BLIQUE-TMI measures the quality of TMIs by considering the following aspects: 1) visual information; 2) local structure; and 3) naturalness. To be specific, quality-aware features related to the former two aspects are extracted in a local manner based on sparse representation, while quality-aware features related to the third aspect are derived based on global statistics modeling in both intensity and color domains. All the extracted local and global quality-aware features constitute a final feature vector. An emergent machine learning technique, i.e., extreme learning machine, is adopted to learn a quality predictor from feature space to quality space. The superiority of BLIQUE-TMI to several leading blind IQA metrics is well demonstrated on two benchmark databases. [ABSTRACT FROM AUTHOR]

The emerging of depth-camera technology is paving the way for a variety of new applications and it is believed that plane detection is one of them. In fact, planes are common in man-made living structures, thus their accurate detection can benefit many visual-based applications. The use of depth information allows detecting planes characterized by complex pattern and texture, where the texture-based plane detection algorithms usually fail. In this paper, we propose a robust depth-driven plane detection (DPD) algorithm which consists of two parts: the growing-based plane detection and a two-stage refinement. The proposed approach starts from the seed patch with the highest planarity and uses the estimated equation of the growing plane and a dynamic threshold function to steer the growing process. Aided with this mechanism, each seed patch can grow to its maximum extent, and then the next seed patch starts to grow. This process is iteratively repeated so as to detect all the planes. Moreover, the refinement is proposed to tackle two common problems suffered by growing-based approaches, the over-growing problem, and the under-growing problem. Validated by extensive experiments, the proposed DPD algorithm is able to accurately detect planes and robust to various testing conditions. In terms of applications, it can be used as the pre-processing step for a variety of applications, such as, planar object recognition, super-resolution of the time-of-flight depth images with intrinsically low resolution. [ABSTRACT FROM AUTHOR]

High-dimensional data contain not only redundancy but also noises produced by the sensors. These noises are usually non-Gaussian distributed. The metrics based on Euclidean distance are not suitable for these situations in general. In order to select the useful features and combat the adverse effects of the noises simultaneously, a robust sparse subspace learning method in unsupervised scenario is proposed in this paper based on the maximum correntropy criterion that shows strong robustness against outliers. Furthermore, an iterative strategy based on half quadratic and an accelerated block coordinate update is proposed. The convergence analysis of the proposed method is also carried out to ensure the convergence to a reliable solution. Extensive experiments are conducted on real-world data sets to show that the new method can filter out the outliers and outperform several state-of-the-art unsupervised feature selection methods. [ABSTRACT FROM AUTHOR]

Frame rate up-conversion (FRUC) is a technology for generating a higher frame rate video from one with a lower frame rate. The multi-view plus depth map (MVD) system has been recently considered a promising solution for three-dimensional video systems. Accordingly, an FRUC efficient solution for the MVD system is greatly needed. In this paper, we thus propose an FRUC approach that considers the MVD system characteristics. It divides a frame into multiple layers using a depth map, and it reconstructs occluded regions using neighboring views and depth map information. With reconstructed information, it accurately performs motion estimation when a region occlusion occurs. Experimental results demonstrated that the proposed method significantly improved the quality of interpolated frames, and it can effectively complement other conventional FRUC approaches. [ABSTRACT FROM AUTHOR]

Objectness has recently become a standard step in many computer vision tasks. Among various techniques, those based on hierarchical image segmentation play a fundamental role for developments in new data modalities. In this paper, we address the problem of objectness in RGB-D images and propose a novel and effective approach, namely, hierarchical image segmentation ensemble (HISE). Different from existing image segmentation based methods that generate object segments or proposals largely by heuristics or empirical rules, HISE learns superpixel mergings with a hierarchical tree-structured ensemble, where individual merging models of the ensemble are formed by traversing different paths of the tree, and where both the merging accuracy and proposal diversity are emphasized. Furthermore, we use efficient feature measurements that support easy integration of additional clues. Extensive experiments conducted on the benchmark NYU-v2 RGB-D and SUN RGB-D data sets show the competency of our proposed method. [ABSTRACT FROM AUTHOR]

A video compression framework based on spatio-temporal resolution adaptation (ViSTRA) is proposed, which dynamically resamples the input video spatially and temporally during encoding, based on a quantisation-resolution decision, and reconstructs the full resolution video at the decoder. Temporal upsampling is performed using frame repetition, whereas a convolutional neural network super-resolution model is employed for spatial resolution upsampling. ViSTRA has been integrated into the high efficiency video coding reference software (HM 16.14). Experimental results verified via an international challenge show significant improvements, with BD-rate gains of 15% based on PSNR and an average MOS difference of 0.5 based on subjective visual quality tests. [ABSTRACT FROM AUTHOR]

Contrast enhancement is a crucial image processing step for image quality control. However, images enhanced by conventional contrast enhancement methods can have negative effects on the performance of image quality control. The most commonly observed effects are over- and under-enhancement effects on images, which cause significant loss of fine textures in images. This paper developed a new contrast enhancement algorithm based on an entropy-preserving mapping prior that improves on conventional contrast enhancement methods. By creating a closed-form solution for enhancing the image contrast under this novel prior and learning the coefficients of the solution using an unsupervised learning strategy, an image’s contrast and texture can be effectively recovered. The experimental results verify that our proposed method clearly outperforms the existing state-of-the-art methods in terms of both quantitative estimation and qualitative human visual inspection. [ABSTRACT FROM AUTHOR]

Rate control is a key technique for video communication systems. The aim of rate control is to transmit the best possible quality video sequences under various restrictions, such as channel bandwidth, buffer capacity, maximum time delay allowed for a given service, and so on. The $\lambda $ domain rate control technique ($\lambda $ -RC) has been integrated into the latest High Efficiency Video Coding standard (H.265/HEVC) test model, due to its accurate bit estimation and high rate-distortion performance. However, it is found that the $\lambda $ -RC is not the optimal choice under the random-access configuration. When the random-access configuration is used, pictures are organized into temporal layers, where pictures in different layers are of different importance in terms of prediction. In this paper, a picture level lambda domain rate control technique for the random-access configuration in H.265/HEVC is proposed. The influence of temporal layers is effectively considered in the proposed algorithm referred to as TL- $\lambda $ -PRC. Experimental results verify that the proposed TL- $\lambda $ -PRC is efficient in coding performance and accurate in bit estimation. Compared with the $\lambda $ -RC with the fixed ration bit allocation which has been implemented in the test model of H.265/HEVC (HM 14.0), TL- $\lambda $ -PRC achieves an average reduction of 4.10% and 3.49% for slow motion and fast motion sequences, respectively, in BD-rate (Bjøntegaard-Delta bitrate) with more accurate bit estimation. The performance of different algorithms in terms of algorithm complexity and quality fluctuation are also carefully analyzed in this contribution. [ABSTRACT FROM AUTHOR]

Recently, multiple ($k, n$) region-incrementing VCSs (RIVCSs) have been proposed that can gradually reconstruct secrets in a single image. In ($k, n$)-RIVCS, the secret image is subdivided into multiple regions in such a way that any $t$ shadow images, where $k \le t \le n$ , can be used to reveal the secret in the ($t-k+1$)th region. This region-incrementing property provides progressive decoding. However, the regions in previous RIVCSs are disjointed: no two regions have overlapping areas. In this paper, we discuss a ($k, n$) region-in-region progressive VCS (RPVCS). Our ($k, n$)-RPVCS has ($n-k+1$) secrecy-level regions, and each region can be located within any previous region. This new type of region allocation not only provides more areas in which to hide the secret compared with the non-overlapping regions in RIVCS but also provides different presentation methods in progressive decoding. [ABSTRACT FROM AUTHOR]

Low-illumination images are usually taken in non-uniform environmental light, such as extremely dark or bright light or artificial light. The enhancement results achieved by existing techniques are prone to halo artifacts, color unnaturalness, and information loss. To address these problems, we present a physical lighting model that describes the degradation of poor illumination images, in which the environmental light is a point-wise variable and changes with the local light source. As long as the parameters in the model are properly estimated, the low-illumination images can be directly recovered by solving the model. First, the initial environmental light can be considered as the incident component according to the Retinex theory and estimated via a Gaussian surrounding function. Second, the environmental light and light-scattering attenuation rate are iteratively adjusted with the information loss constraint. Finally, to restrain the halo and block effects, the two parameters are refined by the weighted guide filter. The experimental results indicate that the proposed algorithm can improve the appearance of low-illumination images that are captured in different scenes, reveal the details in textured regions with few halo effects, increase the richness of the visible edges, retain color consistency and reproduce the color quality and naturalness. [ABSTRACT FROM AUTHOR]

The main challenges of visual object tracking arise from the arbitrary appearance of the objects that need to be tracked. Most existing algorithms try to solve this problem by training a new model to regenerate or classify each tracked object. As a result, the model needs to be initialized and retrained for each new object. In this paper, we propose to track different objects in an object-independent approach with a novel two-flow convolutional neural network (YCNN). The YCNN takes two inputs (one is an object image patch, the other is a larger searching image patch), then outputs a response map which predicts how likely and where the object would appear in the search patch. Unlike the object-specific approaches, the YCNN is actually trained to measure the similarity between the two image patches. Thus, this model will not be limited to any specific object. Furthermore, the network is end-to-end trained to extract both shallow and deep dedicated convolutional features for visual tracking. And once properly trained, the YCNN can be used to track all kinds of objects without further training and updating. As a result, our algorithm is able to run at a very high speed of 45 frames-per-second. The effectiveness of the proposed algorithm can also be proved by the experiments on two popular data sets: OTB-100 and VOT-2014. [ABSTRACT FROM AUTHOR]

Nonnegative matrix factorization (NMF), known as a famous matrix factorization technique, has been widely used in pattern recognition and computer vision. NMF represents the input data matrix as a product of two nonnegative factors. As NMF is based on the Euclidean distance, which is sensitive to noise or errors in the data, some robust NMF methods are proposed. Mainly focusing on parts-based representation, these robust NMF methods often neglect global representation of data. In fact, the global geometry information of data is more robust than the local information about the noisy data in terms of image classification. In order to effectively improve the robustness of NMF and learn part-based and global representation of the data, a novel method low-rank nonnegative factorization (LRNF) is proposed in this paper. First, we assume that the data are grossly corrupted, and the $L_{1} $ norm is used as a sparse constraint on the assumed noise matrix. Then, LRNF learns a low-rank matrix with the global representation ability. Finally, we make a nonnegative factorization of the learned low-rank matrix. We can obtain a base matrix, which preserves locality and globality properties of the data in the meantime. Extensive experiments have been conducted on nine real-world image databases to verify the performance of the proposed LRNF method by comparing with the state-of-the-art algorithms on robust dimensionality reduction. [ABSTRACT FROM AUTHOR]

Context-based adaptive arithmetic coding (CAAC) has high coding efficiency and is adopted by the majority of advanced compression algorithms. In this paper, five new techniques are proposed to further improve the performance of CAAC. They make the frequency table (the table used to estimate the probability distribution of data according to the past input) of CAAC converge to the true probability distribution rapidly and hence improve the coding efficiency. Instead of varying only one entry of the frequency table, the proposed range-adjusting scheme adjusts the entries near to the current input value together. With the proposed mutual-learning scheme, the frequency tables of the contexts highly correlated to the current context are also adjusted. The proposed increasingly adjusting step scheme applies a greater adjusting step for recent data. The proposed adaptive initialization scheme uses a proper model to initialize the frequency table. Moreover, a local frequency table is generated according to local information. We perform several simulations on edge-directed prediction-based lossless image compression, coefficient encoding in JPEG, bit plane coding in JPEG 2000, and motion vector residue coding in video compression. All simulations confirm that the proposed techniques can reduce the bit rate and are beneficial for data compression. [ABSTRACT FROM AUTHOR]

Recently, some picture-embedding schemes have been proposed to improve the aesthetic appearance of 2D barcodes. However, these aesthetic 2D barcodes are not robust to the distortions incurred by the print/display-and-capture channel under limited rendering space and resolution. In this paper, a picture-embedding 2D barcode named the Robust and Aesthetic (RA) Code is proposed to counter the channel impairments, such as downsampling error and inter-symbol interference. Experimental results demonstrate that the proposed RA Code is more robust than the existing picture-embedding 2D barcodes, especially under limited printing/displaying space. The RA Code of a size as small as $1.5 \times 1.5 {\,\,\mbox {cm}}^{2}$ can be successfully decoded with a demodulated bit error probability of less than 0.1, which corresponds to more than 50% improvement over those of the state-of-the-art picture-embedding 2D barcodes. In addition, the correct decoding probability is improved significantly from as low as 0 to more than 0.8650. The decoding algorithm has been implemented on the Android platform, and the practicality of the RA Code has been successfully demonstrated. [ABSTRACT FROM AUTHOR]

Deep learning, e.g., convolutional neural networks (CNNs), has achieved great success in image processing and computer vision especially in high-level vision applications, such as recognition and understanding. However, it is rarely used to solve low-level vision problems such as image compression studied in this paper. Here, we move forward a step and propose a novel compression framework based on CNNs. To achieve high-quality image compression at low bit rates, two CNNs are seamlessly integrated into an end-to-end compression framework. The first CNN, named compact convolutional neural network (ComCNN), learns an optimal compact representation from an input image, which preserves the structural information and is then encoded using an image codec (e.g., JPEG, JPEG2000, or BPG). The second CNN, named reconstruction convolutional neural network (RecCNN), is used to reconstruct the decoded image with high quality in the decoding end. To make two CNNs effectively collaborate, we develop a unified end-to-end learning algorithm to simultaneously learn ComCNN and RecCNN, which facilitates the accurate reconstruction of the decoded image using RecCNN. Such a design also makes the proposed compression framework compatible with existing image coding standards. Experimental results validate that the proposed compression framework greatly outperforms several compression frameworks that use existing image coding standards with the state-of-the-art deblocking or denoising post-processing methods. [ABSTRACT FROM AUTHOR]

Gesture recognition has raised wide attention in computer vision owing to its many applications. However, the task of video-based large-scale gesture recognition yet faces many challenges, since many gesture-irrelevant factors like the background may disturb the recognition accuracy. To better recognize gestures with large-scale videos, we propose a method based on RGB-D data in this paper, where the “RGB-D” means RGB and depth data captured simultaneously by specific devices like Kinect. To learn gesture details better, we first use an adaptive frame unification strategy to unify the frame number of inputs, and then the RGB and depth data are sent to the C3D model to extract spatiotemporal features, respectively. In order to alleviate the interference of gesture-irrelevant factors, the saliency theory is also employed to generate auxiliary data. Next the features of these data are combined to boost the performance, which can also avoid unreasonable synthetic data, since the dimension of C3D features is uniform. Finally the performances of several classifiers are tested and the best one of SVM classifier is selected to output the ultimate accuracy. Our approach achieves 52.04% and 59.43% accuracy on the validation and testing subset of the Chalearn LAP IsoGD, respectively, both of which outperform our results in the chalearn LAP Large-scale Gesture Recognition Challenge as reported in ICPR 2016. [ABSTRACT FROM AUTHOR]

We consider the person re-identification problem, assuming the availability of a sequence of images for each person, commonly referred to as video-based or multi-shot re-identification. We approach this problem from the perspective of learning discriminative distance metric functions. While existing distance metric learning methods typically employ the average feature vector as the data exemplar, this discards the inherent structure of the data. To overcome this issue, we describe the image sequence data using affine hulls. We show that directly computing the distance between the closest points on these affine hulls as in existing recognition algorithms is not sufficiently discriminative in the context of person re-identification. To this end, we incorporate affine hull data modeling into the traditional distance metric learning framework, learning discriminative feature representations directly using affine hulls. We perform extensive experiments on several publicly available data sets to show that the proposed approach improves the performance of existing metric learning algorithms irrespective of the feature space employed to perform metric learning. Furthermore, we advance the state of the art on iLIDS-VID, PRID, and SAIVT, with absolute rank-1 performance improvements of 6.0%, 11.4%, and 6.0% respectively. [ABSTRACT FROM AUTHOR]

Video-based person re-identification (re-id) has attracted a lot of research interest. When facing dramatic growth in new pedestrian videos, existing video-based person re-id methods usually need large quantities of labeled pedestrian videos to train a discriminative model. In practice, labeling large quantities of pedestrian videos is a costly and time-consuming task, which will limit the application of these methods in the real environment. Therefore, it is valuable and necessary to investigate how to learn a discriminative re-id model by using limited labeled training pedestrian videos. In this paper, we propose a semi-supervised cross-view projection-based dictionary learning (SCPDL) approach for video-based person re-id. Specifically, SCPDL jointly learns a pair of feature projection matrices and a pair of dictionaries by integrating the information contained in labeled and unlabeled pedestrian videos. With the learned feature projection matrices, the influence of variations within each video to the re-id can be reduced. With the learned dictionary pair, pedestrian videos from two different cameras can be converted into coding coefficients in a common representation space, such that the differences between different cameras can be bridged. In the learning process, the labeled pedestrian videos are used to ensure that the learned dictionaries have favorable discriminability; the large quantities of unlabeled pedestrian videos are used to ensure that SCPDL can better capture the variations between pedestrian videos, such that the learned dictionaries can own stronger representative capability. Experiments on two public pedestrian sequence data sets (iLIDS-VID and PRID 2011) demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Emotion recognition is challenging due to the emotional gap between emotions and audio–visual features. Motivated by the powerful feature learning ability of deep neural networks, this paper proposes to bridge the emotional gap by using a hybrid deep model, which first produces audio–visual segment features with Convolutional Neural Networks (CNNs) and 3D-CNN, then fuses audio–visual segment features in a Deep Belief Networks (DBNs). The proposed method is trained in two stages. First, CNN and 3D-CNN models pre-trained on corresponding large-scale image and video classification tasks are fine-tuned on emotion recognition tasks to learn audio and visual segment features, respectively. Second, the outputs of CNN and 3D-CNN models are combined into a fusion network built with a DBN model. The fusion network is trained to jointly learn a discriminative audio–visual segment feature representation. After average-pooling segment features learned by DBN to form a fixed-length global video feature, a linear Support Vector Machine is used for video emotion classification. Experimental results on three public audio–visual emotional databases, including the acted RML database, the acted eNTERFACE05 database, and the spontaneous BAUM-1s database, demonstrate the promising performance of the proposed method. To the best of our knowledge, this is an early work fusing audio and visual cues with CNN, 3D-CNN, and DBN for audio–visual emotion recognition. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of recognizing human actions from depth videos. We propose a depth-based local descriptor and affine subspace coding representation with locality-constrained affine subspace coding (LASC) for 3D action recognition. First, each depth video sequence is divided into a set of subsequences (i.e., multi-scale sub-actions) based on the normalized motion energy vector. Next, depth motion map-based gradient local auto-correlation features are employed to capture the shape information and motion cues of each sub-action. In order to obtain discriminative and compact representation, we extract the local high-order information of the depth video using LASC. Through experiments, we show that the use of LASC exhibits better performance compared with existing methods such as locality-constrained linear coding. We compared LASC with the state-of-the-art methods based on similar principle, using features extracted from a single modality, on four datasets, and with those using multiple features or nonlinear recognition machines. The results on four datasets clearly show the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Detection of moving objects in videos is a crucial step toward successful surveillance and monitoring applications. A key component for such tasks is called background subtraction and tries to extract regions of interest from the image background for further processing or action. For this reason, its accuracy and real-time performance are of great significance. Although effective background subtraction methods have been proposed, only a few of them take into consideration the special characteristics of thermal imagery. In this paper, we propose a background subtraction scheme, which models the thermal responses of each pixel as a mixture of Gaussians with unknown number of components. Following a Bayesian approach, our method automatically estimates the mixture structure, while simultaneously it avoids over-/underfitting. The pixel density estimate is followed by an efficient and highly accurate updating mechanism, which permits our system to be automatically adapted to dynamically changing operation conditions. We propose a reference implementation of our method in reconfigurable hardware achieving both adequate performance and low-power consumption. Adopting a high-level synthesis design and demanding floating point arithmetic operations are mapped in reconfigurable hardware, demonstrating fast prototyping and on-field customization at the same time. [ABSTRACT FROM AUTHOR]

In this paper, a novel image quality assessment (IQA) model for the screen content images (SCIs) is proposed by using multi-scale difference of Gaussian (MDOG). Motivated by the observation that the human visual system (HVS) is sensitive to the edges while the image details can be better explored in different scales, the proposed model exploits MDOG to effectively characterize the edge information of the reference and distorted SCIs at two different scales, respectively. Then, the degree of edge similarity is measured in terms of the smaller-scale edge map. Finally, the edge strength computed based on the larger-scale edge map is used as the weighting factor to generate the final SCI quality score. Experimental results have shown that the proposed IQA model for the SCIs produces high consistency with human perception of the SCI quality and outperforms the state-of-the-art quality models. [ABSTRACT FROM AUTHOR]