Search

Image quality assessment (IQA) is an important task of image processing and has diverse applications, such as image super-resolution reconstruction, image transmission and monitoring systems. This paper proposes a perceptual hashing algorithm with complementary color wavelet transform (CCWT) and compressed sensing (CS) for reduced-reference (RR) IQA. The CCWT is exploited to decompose input color image into different sub-bands. Since the calculation of CCWT uses all color channels without discarding any information, the distortions introduced by digital operations on color channels are preserved in the CCWT sub-bands. The block-based CS is used to extract features from the CCWT sub-bands. As the Euclidean distance between the block-based CS features is slightly influenced by content-preserving operations, perceptual features constructed by Euclidean distances are robust, discriminative and compact. Hash sequence is finally determined by quantifying the perceptual features. Effectiveness of the proposed hashing is verified by various experiments on four open image databases. Experimental results demonstrate that the proposed hashing is superior to some state-of-the-art algorithms in terms of classification and RR IQA application. [ABSTRACT FROM AUTHOR]

In this paper, we conduct an extensive study on the use of pre-trained convolutional neural networks (CNNs) for omnidirectional image quality assessment (IQA). To cope with the lack of available IQA databases, transfer learning from seven pre-trained CNN models is investigated over retraining on standard 2D databases. In addition, we explore the influence of various image representations and training strategies on the model’s performance. A comparison of the use of projected versus radial content, and multichannel CNN versus patch-wise training is also covered. The experimental results on two publicly available databases are used to draw conclusions about which strategy best fits the visual quality prediction and at which computational cost. The analysis shows that retraining CNN models on 2D IQA databases improves the prediction accuracy. The latter and the required computational time are found to be significantly affected by the training strategy. Cross-database evaluations demonstrate that the nature and variety of the content impact the generalization ability of the models. Finally, we show that conclusions coming from other image processing communities may not hold for IQA. The provided discussion shall provide insights and recommendations when using pre-trained CNNs for omnidirectional IQA. [ABSTRACT FROM AUTHOR]

Image quality assessment (IQA) plays a central role in many image processing algorithms and systems. Although many popular IQA models achieves high performance on existing released databases, they are still not well accepted in practical applications due to the not-always satisfactory accuracy on real-world data and situations. In this paper, we revisit the IQA research, and point out an ignored but interesting problem in IQA: the coarse-grained (i.e., when quality variation is sufficiently big, as the setting of most IQA databases up to date) statistical results evaluated on existing databases mask the fine-grained differentiation. Accordingly, we present a survey on image quality assessment from a new perspective: fine-grained image quality assessment (FG-IQA). Recent FG-IQA research on five major kinds of images is introduced, and some popular IQA methods are analyzed from FG-IQA perspective. The potential problems for current IQA research based on existing coarse-grained databases are analyzed and the necessity of more FG-IQA research is justified. Finally, we discuss some challenges and possible directions for future works in FG-IQA. [ABSTRACT FROM AUTHOR]

Retrieving 3D shapes based on 2D images is a challenging research topic, due to the significant gap between different domains. Recently, various approaches have been proposed to handle this problem. However, the majority of methods target the cross-domain retrieval task as a pure domain adaptation problem, which focuses on the alignment but ignores the visual relevance between the 2D images and their corresponding 3D shapes. To fundamentally decrease the divergence between different domains, we propose a novel cross-domain learning network (CLN) for 2D image-based 3D shape retrieval task. First, we estimate the pose information from the 2D image to guide the view rendering of 3D shapes, which increases the visual correlations of the cross-domain data to eliminate the divergence between them. Second, we introduce a novel joint learning network, considering both the domain-specific characteristics and the cross-domain interactions for data alignment, which further compensates for the gap between different domains by controlling the distance of intra- and inter-classes. After the metric learning process, discriminative descriptors of images and shapes are generated for the cross-domain retrieval task. To prove the effectiveness and robustness of the proposed method, we conduct extensive experiments on the MI3DOR, SHREC’13, and SHREC’14 datasets. The experimental results demonstrate the superiority of our proposed method, and significant improvements have been achieved compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this work, we simulate the early visual information processing mechanisms in biological visual system (BVS) to solve the visible-infrared image fusion (VIIF) task. Concretely, both infrared and visible images are first processed with a dynamic receptive field (DRF), which is imitated by a Difference of Gaussian (DoG) function whose parameters are adjusted according to local image statistics (e.g., local edge responses). The DRF processing produces two components for each source image (e.g., the visible image or the infrared image) that respectively represent the results of On-center based DRF and Off-center based DRF. Then, the results of On-center based DRF for visible image are fused with the results of Off-center based DRF for infrared image and the results of On-center based DRF for infrared image are fused with the results of Off-center based DRF for visible image, according to the mechanisms of cortex-based center-surround fusion. Algorithmically, this step fuses the visible and infrared images processed by DRF with On-center and Off-center according to the different levels of local homogeneity. Moreover, a feedward signal acting as the sub-cortical flow is also introduced to adjust the results during the fusion. The final output image is simply obtained by the summation of two components after the cortex and sub-cortex based fusion. Qualitative and quantitative tests on four datasets demonstrate that our approach can fuse the infrared and visible images effectively with the good background details and discernible salient areas. We emphasize the importance of corticothalamic feedback, cortex and sub-cortex-based fusion, and the interactions between On-center pathway and Off-center pathway that are ubiquitous in BVS for producing the high-quality visual signals. [ABSTRACT FROM AUTHOR]

It is challenging to extend the support region of state-of-the-art local edge-preserving filtering approaches to the entire input image on account of huge memory cost and heavy computational burden. In this paper, we propose an O(N) time recursive non-local edge-aware filter. A novel graph and a linear time algorithm are presented to effectively propagate information across the entire image. In this graph, information is propagated along all directions to avoid visual artifacts. Both the intensity similarity and the spatial affinity are utilized to estimate the similarity of neighboring pixels, and the distance of any two pixels is the length of the path between these two pixels on a binary tree. Specifically, the input image is filtered at four directions, namely left-to-right, right-to-left, top-to-bottom and bottom-to-top. In each pass, the un-normalized output and the normalization constant of the root node are computed recursively from leaf nodes to the root node on a binary tree in linear time. The filtering output is the average of the outputs for these four directions. A comparison with other non-local edge-aware filters is presented to show the advantages of our approach. Extensive experiments demonstrate that our recursive non-local edge-preserving filter is effective in a variety of computer vision and image processing applications, including edge-aware filtering, detail enhancement, stylization, and stereo matching. [ABSTRACT FROM AUTHOR]

In this paper, we present a comparative evaluation of various multi-person tracking-by-detection approaches on public data sets. This paper investigates five popular trackers coupled with six relevant visual people detectors evaluated on seven public data sets. The evaluation emphasizes on exhibited performance variation depending on tracker-detector choices. Our experimental results show that the overall performance depends on how challenging the data set is, the performance of the detector on the specific data set, and the tracker-detector combination. Some trackers are more sensitive to the choice of a detector and some detectors to the choice of a tracker than others. Based on our results, two of the trackers demonstrate the best performances consistently across different data sets, whereas the best performing detectors vary per data set. This underscores the need for careful application context specific evaluation when choosing a detector. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel deep learning-based visual comfort assessment (VCA) for stereoscopic images. To assess the overall degree of visual discomfort in stereoscopic viewing, we devise a binocular fusion deep network (BFN) learning binocular characteristics between stereoscopic images. The proposed BFN learns the latent binocular feature representations for the visual comfort score prediction. In the BFN, the binocular feature is encoded by fusing the spatial features extracted from left and right views. Finally, the visual comfort score is predicted by projecting the binocular feature onto the subjective score space. In addition, we devise a disparity regularization network (DRN) for improving the prediction results. The proposed DRN takes the binocular feature from the BFN and estimates disparity maps from the feature in order to embed disparity relations between left and right views into the deep network. The proposed deep network with BFN and DRN is end-to-end trained in a unified framework in which the DRN acts as disparity regularization. We evaluated the prediction performance of the proposed deep network for VCA by the comparison of existing objective VCA metrics. Further, we demonstrated that the proposed BFN showed various factors causing visual discomfort by using network visualization. [ABSTRACT FROM AUTHOR]

High dynamic range (HDR) image, which has a powerful capacity to represent the wide dynamic range of real-world scenes, has been receiving attention from both academic and industrial communities. Although HDR imaging devices have become prevalent, the display devices for HDR images are still limited. To facilitate the visualization of HDR images in standard low dynamic range displays, many different tone mapping operators (TMOs) have been developed. To create a fair comparison of different TMOs, this paper proposes a BLInd QUality Evaluator to blindly predict the quality of Tone-Mapped Images (BLIQUE-TMI) without accessing the corresponding HDR versions. BLIQUE-TMI measures the quality of TMIs by considering the following aspects: 1) visual information; 2) local structure; and 3) naturalness. To be specific, quality-aware features related to the former two aspects are extracted in a local manner based on sparse representation, while quality-aware features related to the third aspect are derived based on global statistics modeling in both intensity and color domains. All the extracted local and global quality-aware features constitute a final feature vector. An emergent machine learning technique, i.e., extreme learning machine, is adopted to learn a quality predictor from feature space to quality space. The superiority of BLIQUE-TMI to several leading blind IQA metrics is well demonstrated on two benchmark databases. [ABSTRACT FROM AUTHOR]

Recently, multiple ($k, n$) region-incrementing VCSs (RIVCSs) have been proposed that can gradually reconstruct secrets in a single image. In ($k, n$)-RIVCS, the secret image is subdivided into multiple regions in such a way that any $t$ shadow images, where $k \le t \le n$ , can be used to reveal the secret in the ($t-k+1$)th region. This region-incrementing property provides progressive decoding. However, the regions in previous RIVCSs are disjointed: no two regions have overlapping areas. In this paper, we discuss a ($k, n$) region-in-region progressive VCS (RPVCS). Our ($k, n$)-RPVCS has ($n-k+1$) secrecy-level regions, and each region can be located within any previous region. This new type of region allocation not only provides more areas in which to hide the secret compared with the non-overlapping regions in RIVCS but also provides different presentation methods in progressive decoding. [ABSTRACT FROM AUTHOR]

The main challenges of visual object tracking arise from the arbitrary appearance of the objects that need to be tracked. Most existing algorithms try to solve this problem by training a new model to regenerate or classify each tracked object. As a result, the model needs to be initialized and retrained for each new object. In this paper, we propose to track different objects in an object-independent approach with a novel two-flow convolutional neural network (YCNN). The YCNN takes two inputs (one is an object image patch, the other is a larger searching image patch), then outputs a response map which predicts how likely and where the object would appear in the search patch. Unlike the object-specific approaches, the YCNN is actually trained to measure the similarity between the two image patches. Thus, this model will not be limited to any specific object. Furthermore, the network is end-to-end trained to extract both shallow and deep dedicated convolutional features for visual tracking. And once properly trained, the YCNN can be used to track all kinds of objects without further training and updating. As a result, our algorithm is able to run at a very high speed of 45 frames-per-second. The effectiveness of the proposed algorithm can also be proved by the experiments on two popular data sets: OTB-100 and VOT-2014. [ABSTRACT FROM AUTHOR]

Emotion recognition is challenging due to the emotional gap between emotions and audio–visual features. Motivated by the powerful feature learning ability of deep neural networks, this paper proposes to bridge the emotional gap by using a hybrid deep model, which first produces audio–visual segment features with Convolutional Neural Networks (CNNs) and 3D-CNN, then fuses audio–visual segment features in a Deep Belief Networks (DBNs). The proposed method is trained in two stages. First, CNN and 3D-CNN models pre-trained on corresponding large-scale image and video classification tasks are fine-tuned on emotion recognition tasks to learn audio and visual segment features, respectively. Second, the outputs of CNN and 3D-CNN models are combined into a fusion network built with a DBN model. The fusion network is trained to jointly learn a discriminative audio–visual segment feature representation. After average-pooling segment features learned by DBN to form a fixed-length global video feature, a linear Support Vector Machine is used for video emotion classification. Experimental results on three public audio–visual emotional databases, including the acted RML database, the acted eNTERFACE05 database, and the spontaneous BAUM-1s database, demonstrate the promising performance of the proposed method. To the best of our knowledge, this is an early work fusing audio and visual cues with CNN, 3D-CNN, and DBN for audio–visual emotion recognition. [ABSTRACT FROM AUTHOR]

Individuating and locating repetitive patterns in still images is a fundamental task in image processing, typically achieved by means of correlation strategies. In this paper, we provide a solid solution to this task using a differential geometry approach, operating on Lie algebra, and exploiting a mixture of templates. The proposed method asks the user to locate a few instances of the target patterns (seeds) that become visual templates used to explore the image. We propose an iterative algorithm to locate patches similar to the seeds working in three steps: first, clustering the detected patches to generate templates of different classes, then looking for the affine transformations, living on a Lie algebra that best links the templates and the detected patches, and finally detecting new patches with a convolutional strategy. The process ends when no new patches are found. We will show how our method is able to process heterogeneous unstructured images with multiple visual motifs and extremely crowded scenarios with high precision and recall, outperforming all the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Image quality in mobile displays is considerably influenced by ambient light. In the daylight condition, images on mobile displays are darkly perceived by the human visual system due to the limited dynamic range of display, which causes loss of luminance and details. In this paper, we propose readability enhancement of displayed images under ambient light by enhancing both luminance and details. We design a weighted optimization framework, which contains the data term for luminance enhancement and the gradient term for detail enhancement. In the data term, we use an ambient light nonlinear intensity-transfer function considering display properties, ambient light, and image contents. In the gradient term, we employ a threshold versus intensity adaptation function based on the degree of ambient light adaptation to compensate for gradient distortions. We solve the optimization framework using a numerical solver and achieve image enhancement in displayed images. Experimental results demonstrate that the proposed method significantly improves readability of mobile displays under ambient light by enhancing luminance and details of images. [ABSTRACT FROM AUTHOR]

In this paper, we propose a nonlinear structural hashing approach to learn compact binary codes for scalable video search. Unlike most existing video hashing methods which consider image frames within a video separately for binary code learning, we develop a multi-layer neural network to learn compact and discriminative binary codes by exploiting both the structural information between different frames within a video and the nonlinear relationship between video samples. To be specific, we learn these binary codes under two different constraints at the output of our network: 1) the distance between the learned binary codes for frames within the same scene is minimized and 2) the distance between the learned binary matrices for a video pair with the same label is less than a threshold and that for a video pair with different labels is larger than a threshold. To better measure the structural information of the scenes from videos, we employ a subspace clustering method to cluster frames into different scenes. Moreover, we design multiple hierarchical nonlinear transformations to preserve the nonlinear relationship between videos. Experimental results on three video data sets show that our method outperforms state-of-the-art hashing approaches on the scalable video search task. [ABSTRACT FROM AUTHOR]

Retrieving videos from large repositories using image queries is important for many applications, such as brand monitoring or content linking. We introduce a new retrieval architecture, in which the image query can be compared directly with database videos—significantly improving retrieval scalability compared with a baseline system that searches the database on a video frame level. Matching an image to a video is an inherently asymmetric problem. We propose an asymmetric comparison technique for Fisher vectors and systematically explore query or database items with varying amounts of clutter, showing the benefits of the proposed technique. We then propose novel video descriptors that can be compared directly with image descriptors. We start by constructing Fisher vectors for video segments, by exploring different aggregation techniques. For a database of lecture videos, such methods obtain a two orders of magnitude compression gain with respect to a frame-based scheme, with no loss in retrieval accuracy. Then, we consider the design of video descriptors, which combine Fisher embedding with hashing techniques, in a flexible framework based on Bloom filters. Large-scale experiments using three datasets show that this technique enables faster and more memory-efficient retrieval, compared with a frame-based method, with similar accuracy. The proposed techniques are further compared against pre-trained convolutional neural network features, outperforming them on three datasets by a substantial margin. [ABSTRACT FROM AUTHOR]

Objective image quality metrics (IQMs) potentially benefit from the addition of visual saliency. However, challenges to optimizing the performance of saliency-based IQMs remain. A previous eye-tracking study has shown that gaze is concentrated in fewer places in images with highly salient features than in images lacking salient features. From this, it can be inferred that the former are more likely to benefit from adding a saliency term to an IQM. To understand whether these ideas still hold when using computational saliency instead of eye-tracking data, we first conducted a statistical evaluation using 15 state-of-the-art saliency models and 10 well-known IQMs. We then used the results to devise an algorithm, which adaptively incorporates saliency in IQMs for natural scenes, based on saliency dispersion. Experimental results demonstrate that this can give significant improvements. [ABSTRACT FROM AUTHOR]

The screen content images (SCIs) quality influences the user experience and the interactive performance of remote computing systems. With numerous approaches proposed to evaluate the quality of natural images, much less work has been dedicated to reduced-reference image quality assessment (RR-IQA) of SCIs. Here, we propose an RR-IQA method from the perspective of SCI visual perception. In particular, the quality of the distorted SCI is evaluated by comparing a set of extracted statistical features that consider both primary visual information and unpredictable uncertainty. A unique property that differentiates the proposed method from previous RR-IQA methods for natural images is the consideration of behaviors when human subjects view the screen content, which motivates us to establish the perceptual model according to the distinct properties of SCIs. Validations based on the screen content IQA database show that the proposed algorithm provides accurate predictions across a wide range of SCI distortions with negligible transmission overhead. [ABSTRACT FROM PUBLISHER]

In this paper, we present a discriminative tracking method based on dictionary learning and support vector machine (SVM) classification, where the dictionary and the classifier are jointly learned within a unified objective function. A discriminative differential tracking method is proposed, which estimates the motion parameters iteratively by the gradient-based method to maximize the SVM classification score, leading the bounding box to move purposively. As the target appearance may change across frames, an online update scheme is exploited, which not only reserves the discriminative information, but also adaptively accounts for the appearance changes in the dynamic scenes. We examine the proposed method on the benchmark challenging image sequences, including heavy occlusion, pose change, illumination variation, and so on. Extensive evaluations demonstrate that the proposed tracker performs favorably against other state-of-the-art algorithms. [ABSTRACT FROM PUBLISHER]

Content-based image retrieval (CBIR) shows great significance in clinical decision-making, which explores the visual content of medical images rather than keywords, tags, or descriptions. It provides doctors an image-guided approach to explore relevant cases that could offer doctors instructive reference. Mammogram screening has been known to be widely used in the early stage diagnosis of breast cancer and could reduce its morbidity and mortality. In this paper, we aim to develop a scalable CBIR method for a large repository of mammogram. To this end, we extend the original Anchor Graph Hashing (AGH) and propose a new unsupervised hashing algorithm, named as composite AGH with iterative quantization (C-AGH-ITQ), which compresses mammographic regions of interest (ROIs) into compact binary codes and enables real-time searching in Hamming space. Multimodal features and different distance metrics are integrated, performing upon a composite Anchor Graph. To improve the effectiveness of the hash code, quantization error is further iteratively minimized by introducing an orthogonal rotation matrix. We evaluate the presented C-AGH-ITQ algorithm on a data set of 11 533 mammographic ROIs obtained from the Digital Database for Screening Mammography. Our method obtains more than 84% retrieval precision and 93% classification accuracy (using $k$ NN prediction), which demonstrates that hash codes produced by C-AGH-ITQ well capture the visual similarities between mammographic images. In addition, since C-AGH-ITQ ensures linear complexity of the training procedure and constant time for query, our system is readily applicable to large-scale mammogram databases and has the potential to provide abundant clinical cases as reference. [ABSTRACT FROM AUTHOR]

Nonoverlapping multicamera visual object tracking typically consists of two steps: single-camera object tracking (SCT) and inter-camera object tracking (ICT). Most of tracking methods focus on SCT, which happens in the same scene, while for real surveillance scenes, ICT is needed and single-camera tracking methods cannot work effectively. In this paper, we try to improve the overall multicamera object tracking performance by a global graph model with an improved similarity metric. Our method treats the similarities of single-camera tracking and inter-camera tracking differently and obtains the optimization in a global graph model. The results show that our method can work better even in the condition of poor SCT. [ABSTRACT FROM AUTHOR]

The use of local features for image representation has become popular in recent years. Local features are often used in the bag-of-visual-words scheme. Although proven effective, this method still has two drawbacks. First, local regions from which local features are extracted are not discriminative enough for visual tasks. Hence, the combination of local features is necessary. Second, the semantic gap between visual features and human perception also hinders the performance. To address these two problems, in this paper, we propose a novel contextual exemplar classifier-based method for image representation and apply it for classification tasks. Each exemplar classifier is trained to separate one training image from the other images of different classes. We partition each image into a number of regions and use the responses of these exemplar classifiers as the image region’s representation. The contextual relationship is then modeled using mixture Dirichlet distributions. A bilayer model is used to predict image classes with L2 constraints. Experimental results on the Natural Scene, Caltech-101/256, Flower-17/102, and SUN-397 data sets show that the proposed method is able to outperform the state-of-the-art local feature-based methods for image classification. [ABSTRACT FROM PUBLISHER]

This paper presents a novel formulation of video stabilization by directly solving for optimal image warps toward stabilized sequence. With the estimated shaky motion via long or short feature trajectories, our approach encodes another two steps, motion compensation and image warping, into a single global optimization process, rather than operating as two individual steps. This process is done only with positions of embedded mesh vertices as common variables. Spatial and temporal coherence is therein reformulated with similarity-invariant representation of motion trajectories and intra- (and inter-) frame consistency of similar transformations with respect to mesh vertices. Such a one-shot formulation converts video stabilization into a quadratic energy minimization problem defined for image warps, and thus can be efficiently resolved by using a robust solver for sparse linear systems. Experimental results demonstrate the flexibility and efficiency of our approach in producing visually plausible stabilization effects on a variety of videos. [ABSTRACT FROM AUTHOR]

Screen content images are originally captured in a full-chroma format. The chroma downsampling, which is commonly applied to the chroma component in screen content image representation and processing (e.g., YUV4:2:0 compression), will significantly degrade the image quality and create annoying artifacts such as blur and color shifting. To tackle this problem, in this paper we propose luma aware chroma downsampling and upsampling algorithms to jointly improve the quality of the chroma image reconstruction. Guided by the luma information, the chroma upsampling algorithm is proposed with the utilization of major color and index map representation. The geometric information-based linear mapping is developed to transfer the structure of luma to the interpolated chroma. Subsequently, the error sensitivity of the upsampling method is analyzed, and content dependent downsampling algorithm is presented to minimize the error sensitivity function. We further explore the applicability of the proposed scheme in the scenario of screen content compression, targeting at improving the decoded chroma image quality for display. Extensive experimental results demonstrate the viability and efficiency of the proposed scheme. [ABSTRACT FROM AUTHOR]

Binary image descriptors, which derive image feature description from the local image patches directly, are widely adopted in the mobile and embedded applications due to lower computational complexity and memory requirement. With the aim of improving the computation efficiency without degrading recognition performance, a lightweight binary robust descriptor is proposed based on the analysis of the state-of-the-art binary descriptors in this paper. A directional edge detection and optimized keypoint score function are developed to refine the keypoints. In addition, rotation invariance is achieved by executing circular symmetric-based descriptor generation and a coarse-grained orientation calculation method concurrently. The experimental results demonstrate that the proposed keypoint detector and binary descriptor achieve more than two times speedup and at least 23.6% improvement in processing speed with comparable performance, respectively. Furthermore, a very large scale integration architecture is also designed based on in-depth exploration of bit-level and task-level parallelism. Based on the postlayout simulation in a TSMC 65-nm CMOS process, the accelerator can achieve 135 frames/s on 1080p image while only consuming 87.5 mW at a 200-MHz operating frequency. [ABSTRACT FROM PUBLISHER]

It is known that the human visual system (HVS) employs independent processes (distortion detection and artifact perception—also often referred to as near-threshold and suprathreshold distortion perception) to assess video quality for various distortion levels. Visual masking effects also play an important role in video distortion perception, especially within spatial and temporal textures. In this paper, a novel perception-based hybrid model for video quality assessment is presented. This simulates the HVS perception process by adaptively combining noticeable distortion and blurring artifacts using an enhanced nonlinear model. Noticeable distortion is defined by thresholding absolute differences using spatial and temporal tolerance maps that characterize texture masking effects, and this makes a significant contribution to quality assessment when the quality of the distorted video is similar to that of the original video. Characterization of blurring artifacts, estimated by computing high frequency energy variations and weighted with motion speed, is found to further improve metric performance. This is especially true for low quality cases. All stages of our model exploit the orientation selectivity and shift invariance properties of the dual-tree complex wavelet transform. This not only helps to improve the performance but also offers the potential for new low complexity in-loop application. Our approach is evaluated on both the Video Quality Experts Group (VQEG) full reference television Phase I and the Laboratory for Image and Video Engineering (LIVE) video databases. The resulting overall performance is superior to the existing metrics, exhibiting statistically better or equivalent performance with significantly lower complexity. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts. [ABSTRACT FROM PUBLISHER]

In this paper, we propose an efficient blind image quality assessment (BIQA) algorithm, which is characterized by a new feature fusion scheme and a k -nearest-neighbor (KNN)-based quality prediction model. Our goal is to predict the perceptual quality of an image without any prior information of its reference image and distortion type. Since the reference image is inaccessible in many applications, the BIQA is quite desirable in this context. In our method, a new feature fusion scheme is first introduced by combining an image’s statistical information from multiple domains (i.e., discrete cosine transform, wavelet, and spatial domains) and multiple color channels (i.e., Y, Cb, and Cr). Then, the predicted image quality is generated from a nonparametric model, which is referred to as the label transfer (LT). Based on the assumption that similar images share similar perceptual qualities, we implement the LT with an image retrieval procedure, where a query image’s KNNs are searched for from some annotated images. The weighted average of the KNN labels (e.g., difference mean opinion score or mean opinion score) is used as the predicted quality score. The proposed method is straightforward and computationally appealing. Experimental results on three publicly available databases (i.e., LIVE II, TID2008, and CSIQ) show that the proposed method is highly consistent with human perception and outperforms many representative BIQA metrics. [ABSTRACT FROM AUTHOR]

Given a secret image S , a set \mathscr {P} of n participants and a strong access structure ( \Gamma \mathrm {\mathbf {Qual}} , \Gamma \mathrm {\mathbf {Forb}}) , a visual cryptographic scheme (VCS) for general access structures (GVCS) encodes S into n shares of transparencies such that the participants of each qualified set in \Gamma \mathrm {\mathbf {Qual}} can reveal S by superimposing their shares; whereas those of any forbidden set in \Gamma \mathrm {\mathbf {Forb}} obtain nothing about $S$ . Elegant GVCS constructions have been designed with smaller pixel expansions. Yet, whether the pixel expansion derived is minimized is still unknown. In this research, we generalize and extend our recent study, in which the modeling of minimizing the pixel expansion for a ( $k$ , $n$ )-VCS into an integer linear program (ILP) was proposed, to ensure that the constraints for GVCS can be satisfied. The pixel expansion of a GVCS can thus be minimized by solving the corresponding ILP. This is the first result in the literature for acquiring the optimal solution to a GVCS. The computational study demonstrates the effectiveness of our ILP and also verifies that the best solution from previous GVCS approaches is optimal for all strong access structures of $n \le 4$ ; but no more reliable for those of $n\ge 5$ . [ABSTRACT FROM PUBLISHER]

Image tag refinement is the task of refining initial tags of an image such that the refined tags can better reflect the content of the image and, therefore, can help users better access that image. The quality of tag refinement depends on the quality of concept representations that build a mapping from concepts to visual images. While good progress was made in the past decade on tag refinement, the previous approaches only achieved a limited success due to their limited concept representations. In this paper, we show that the visual appearances of a concept consist of both a generic view and a specific view, and therefore we can comprehensively represent a concept by two components. To ensure a clean concept representation, this representation is learned on clean click-through data, where noises are greatly reduced. In the framework, a coarse-to-fine image tag refinement is proposed, which: 1) first generates an efficient star graph to find candidate tags but missing in the initial tag list of an input image and 2) guided by this view-dependent concept representation, formulates a probabilistic objective function to eliminate irrelevant tags. Extensive experiments on two widely used standard data sets (MIRFlickr-25K and NUS-WIDE-270K) demonstrate the effectiveness of our approach. [ABSTRACT FROM PUBLISHER]

This paper investigates a general framework to discover categories of unlabeled scene images according to their appearances (i.e., textures and structures). We jointly solve the two coupled tasks in an unsupervised manner: 1) classifying images without predetermining the number of categories and 2) pursuing generative model for each category. In our method, each image is represented by two types of image descriptors that are effective to capture image appearances from different aspects. By treating each image as a graph vertex, we build up a graph and pose the image categorization as a graph partition process. Specifically, a partitioned subgraph can be regarded as a category of scenes and we define the probabilistic model of graph partition by accumulating the generative models of all separated categories. For efficient inference with the graph, we employ a stochastic cluster sampling algorithm, which is designed based on the Metropolis–Hasting mechanism. During the iterations of inference, the model of each category is analytically updated by a generative learning algorithm. In the experiments, our approach is validated on several challenging databases, and it outperforms other popular state-of-the-art methods. The implementation details and empirical analysis are presented as well. [ABSTRACT FROM PUBLISHER]

There is a dearth of information on how perceived auditory information guides image-viewing behavior. To investigate auditory-driven visual attention, we first generated a human eye-fixation database from a pool of 200 static images and 400 image-audio pairs viewed by 48 subjects. The eye tracking data for the image-audio pairs were captured while participants viewed images, which took place immediately after exposure to coherent/incoherent audio samples. The database was analyzed in terms of time to first fixation, fixation durations on the target object, entropy, AUC, and saliency ratio. It was found that coherent audio information is an important cue for enhancing the feature-specific response to the target object. Conversely, incoherent audio information attenuates this response. Finally, a system predicting the image-viewing with the influence of different audio sources was developed. The detailedly discussed top-down module in the system is composed of auditory estimation based on Gaussian mixture model-maximum a posteriori algorithm-universal background model structure, as well as visual estimation based on the conditional random field model and sparse latent variables. The evaluation experiments show that the proposed models in the system exhibit strong consistency with eye fixations. [ABSTRACT FROM PUBLISHER]

Saliency detection has been extensively studied due to its promising contributions for various computer vision applications. However, most existing methods are easily biased toward edges or corners, which are statistically significant, but not necessarily relevant. Moreover, they often fail to find salient regions in complex scenes due to ambiguities between salient regions and highly textured backgrounds. In this paper, we present a novel unified framework for spatiotemporal saliency detection based on textural contrast. Our method is simple and robust, yet biologically plausible; thus, it can be easily extended to various applications, such as image retargeting, object segmentation, and video surveillance. Based on various datasets, we conduct comparative evaluations of 12 representative saliency detection models presented in the literature, and the results show that the proposed scheme outperforms other previously developed methods in detecting salient regions of the static and dynamic scenes. [ABSTRACT FROM AUTHOR]

Detecting and matching image features is a fundamental task in video analytics and computer vision systems. It establishes the correspondences between two images taken at different time instants or from different viewpoints. However, its large computational complexity has been a challenge to most embedded systems. This paper proposes a new FPGA-based embedded system architecture for feature detection and matching. It consists of scale-invariant feature transform (SIFT) feature detection, as well as binary robust independent elementary features (BRIEF) feature description and matching. It is able to establish accurate correspondences between consecutive frames for 720-p (1280x720) video. It optimizes the FPGA architecture for the SIFT feature detection to reduce the utilization of FPGA resources. Moreover, it implements the BRIEF feature description and matching on FPGA. Due to these contributions, the proposed system achieves feature detection and matching at 60 frame/s for 720-p video. Its processing speed can meet and even exceed the demand of most real-life real-time video analytics applications. Extensive experiments have demonstrated its efficiency and effectiveness. [ABSTRACT FROM PUBLISHER]

Saliency detection is widely used to extract regions of interest in images for various image processing applications. Recently, many saliency detection models have been proposed for video in uncompressed (pixel) domain. However, video over Internet is always stored in compressed domains, such as MPEG2, H.264, and MPEG4 Visual. In this paper, we propose a novel video saliency detection model based on feature contrast in compressed domain. Four types of features including luminance, color, texture, and motion are extracted from the discrete cosine transform coefficients and motion vectors in video bitstream. The static saliency map of unpredicted frames (I frames) is calculated on the basis of luminance, color, and texture features, while the motion saliency map of predicted frames (P and B frames) is computed by motion feature. A new fusion method is designed to combine the static saliency and motion saliency maps to get the final saliency map for each video frame. Due to the directly derived features in compressed domain, the proposed model can predict the salient regions efficiently for video frames. Experimental results on a public database show superior performance of the proposed video saliency detection model in compressed domain. [ABSTRACT FROM AUTHOR]

Automatic detection of salient objects in visual media (e.g., videos and images) has been attracting much attention. The detected salient objects can be utilized for segmentation, recognition, and retrieval. However, the accuracy of saliency detection remains a challenge. The reason behind this challenge is mainly due to the lack of a well-defined model for interpreting saliency formulation. To tackle this problem, this letter proposes to detect salient objects based on selective contrast. Selective contrast intrinsically explores the most distinguishable component information in color, texture, and location. A large number of experiments are thereafter carried out upon a benchmark dataset, and the results are compared with those of 12 other popular state-of-the-art algorithms. In addition, the advantage of the proposed algorithm is also demonstrated in a retargeting application. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel prediction-based reversible steganographic scheme based on image inpainting. First, reference pixels are chosen adaptively according to the distribution characteristics of the image content. Then, the image inpainting technique based on partial differential equations is introduced to generate a prediction image that has similar structural and geometric information as the cover image. Finally, by using the two selected groups of peak points and zero points, the histogram of the prediction error is shifted to embed the secret bits reversibly. Since the same reference pixels can be exploited in the extraction procedure, the embedded secret bits can be extracted from the stego image correctly, and the cover image can be restored losslessly. Through the use of the adaptive strategy for choosing reference pixels and the inpainting predictor, the prediction accuracy is high, and more embeddable pixels are acquired. Thus, the proposed scheme provides a greater embedding rate and better visual quality compared with recently reported methods. [ABSTRACT FROM AUTHOR]

A well-known problem in stereoscopic videos is visual fatigue. However, conventional depth adjustment methods provide little guidance in deciding the amount of depth control or determining the condition for depth control. We propose a depth adjustment method based on binocular fusion characteristics, where the fusion time is used as the parameter for adjustment. Visual comfort enhancement is implemented with the horizontal image shift approach for depth adjustment. The speeded-up robust feature is used to estimate maximum disparity and disparity distribution, while a face detection algorithm is used to estimate the viewing distance with a single web camera. Binocular fusion characteristics are acquired in advance by measuring the time required for fusion under various conditions, including foreground disparity, background disparity, and focal distance for random-dot stereograms. Finally, a subjective evaluation is conducted in fixed and free-to-move viewing conditions, and the results show that comfortable videos were generated based on the proposed depth adjustment method. [ABSTRACT FROM PUBLISHER]

This paper re-examines the problem of visual secret sharing for general access structures by using visual cryptograms of random grids (VCRG). Given a binary or color secret image shared by a set of n participants with a strong access structure, we devise two effective algorithms to produce a set of VCRG so that the members in each qualified set can reconstruct the secret image by superimposing their shares, while those in any forbidden set cannot. Our algorithms do not require any extra pixel expansion, which is indispensable and grows exponentially as n increases in conventional visual cryptographic schemes. The feasibility, light contrasts, flexibility, and limitations of our algorithms are explored from both theoretical and empirical points of view. [ABSTRACT FROM PUBLISHER]

Computer screens contain discontinuous-tone content and continuous-tone content. Thus, the most effective way for screen content coding (SCC) is to use two essentially different coders: a dictionary-entropy coder and a traditional hybrid coder. Although screen content is originally in a full-chroma (e.g., YUV444) format, the current method of compression is to first subsample chroma of pictures and then compress pictures using a chroma-subsampled (e.g., YUV420) coder. Using two chroma-subsampled coders cannot achieve high-quality SCC, but using two full-chroma coders is overkill and inefficient for SCC. To solve the dilemma, this paper proposes a mixed chroma sampling-rate approach for SCC. An original full-chroma input macroblock (coding unit) or its prediction residual is chroma-subsampled. One full-chroma base coder and one chroma-subsampled base coder are used simultaneously to code the original and the chroma-subsampled macroblock, respectively. The coder minimizing rate-distortion (R-D) is selected as the final coder for the macroblock. The two base coders are coherently unified and optimized to get the best overall coding performance and share coding components and resources as much as possible. The approach achieves very high visual quality with minimal computing complexity increment for SCC, and has better R-D performance than two full-chroma coders approach, especially in low bitrate. [ABSTRACT FROM PUBLISHER]

Recently, Wang introduced a novel (2, n) region incrementing visual cryptographic scheme (RIVCS), which can gradually reconstruct secrets in a single image with multiple security levels. In RIVCS, the secret image is subdivided into multiple regions in such a way that any t shadow images, where 2\leq t\leq n, can be used to reveal the (t-1)th region. However, Wang's scheme suffers from the incorrect-color problem, which the colors of reconstructed images may be reversed (i.e., the black and white are reversed). If the color of text is also the secret information, the incorrect-color problem will compromise the secret. Additionally, Wang's scheme is only suitable for the 2-out-of-n case, i.e., (k,n)-RIVCS where k=2. In this paper, we propose a general (k,n)-RIVCS, where k and n are any integers, that is able to reveal correct colors of all regions. This paper has made three main contributions: 1) our scheme is a general (k,n)-RIVCS, where k and n can be any integers; 2) the incorrect-color problem is solved; and 3) our (k,n)-RIVCS is theoretically proven to satisfy the security and contrast conditions. [ABSTRACT FROM PUBLISHER]

In just noticeable difference (JND) models, evaluation of contrast masking (CM) is a crucial step. More specifically, CM due to edge masking (EM) and texture masking (TM) needs to be distinguished due to the entropy masking property of the human visual system. However, TM is not estimated accurately in the existing JND models since they fail to distinguish TM from EM. In this letter, we propose an enhanced pixel domain JND model with a new algorithm for CM estimation. In our model, total-variation based image decomposition is used to decompose an image into structural image (i.e., cartoon like, piecewise smooth regions with sharp edges) and textural image for estimation of EM and TM, respectively. Compared with the existing models, the proposed one shows its advantages brought by the better EM and TM estimation. It has been also applied to noise shaping and visual distortion gauge, and favorable results are demonstrated by experiments on different images. [ABSTRACT FROM PUBLISHER]