Search

In this paper, we propose an efficient blind image quality assessment (BIQA) algorithm, which is characterized by a new feature fusion scheme and a k -nearest-neighbor (KNN)-based quality prediction model. Our goal is to predict the perceptual quality of an image without any prior information of its reference image and distortion type. Since the reference image is inaccessible in many applications, the BIQA is quite desirable in this context. In our method, a new feature fusion scheme is first introduced by combining an image’s statistical information from multiple domains (i.e., discrete cosine transform, wavelet, and spatial domains) and multiple color channels (i.e., Y, Cb, and Cr). Then, the predicted image quality is generated from a nonparametric model, which is referred to as the label transfer (LT). Based on the assumption that similar images share similar perceptual qualities, we implement the LT with an image retrieval procedure, where a query image’s KNNs are searched for from some annotated images. The weighted average of the KNN labels (e.g., difference mean opinion score or mean opinion score) is used as the predicted quality score. The proposed method is straightforward and computationally appealing. Experimental results on three publicly available databases (i.e., LIVE II, TID2008, and CSIQ) show that the proposed method is highly consistent with human perception and outperforms many representative BIQA metrics. [ABSTRACT FROM AUTHOR]

To study underlying possibilities for the successes of conventional image representation and deep neural networks (DNNs) in image representation, we propose a deep feature extraction, encoding, and pooling network (DEFEATnet) architecture, which is a marriage between conventional image representation approaches and DNNs. In particular, in DEFEATnet, each layer consists of three components: feature extraction, feature encoding, and pooling. The primary advantage of DEFEATnet is twofold. First, it consolidates the prior knowledge (e.g., translation invariance) from extracting, encoding, and pooling handcrafted features, as in the conventional feature representation approaches. Second, it represents the object parts at different granularities by gradually increasing the local receptive fields in different layers, as in DNNs. Moreover, DEFEATnet is a generalized framework that can readily incorporate all types of local features as well as all kinds of well-designed feature encoding and pooling methods. Since prior knowledge is preserved in DEFEATnet, it is especially useful for image representation on small/medium size data sets, where DNNs usually fail due to the lack of sufficient training data. Promising experimental results clearly show that DEFEATnets outperform shallow conventional image representation approaches by a large margin when the same type of features, feature encoding and pooling are used. The extensive experiments also demonstrate the effectiveness of the deep architecture of our DEFEATnet in improving the robustness for image presentation. [ABSTRACT FROM AUTHOR]

Texture image classification is important in computer vision research. To effectively capture texture patterns, a distinctive feature such as a local binary pattern (LBP) is needed. An LBP is robust against monotonic and gray-scale variations and it computes quickly. Its robustness and speed advantage have made it popular in various texture analysis applications. However, an LBP is sensitive to noise, particularly smooth weak illumination gradients in near-uniform regions. To mitigate the effect of noise and increase distinctiveness, a local ternary pattern (LTP) is proposed. Compared with a binary coding LBP, an LTP adopts ternary coding. As a result, an LTP can better tolerate noise and is significantly more distinctive. These advantages of an LTP effectively improve its classification accuracy. However, the potential of ternary coding is not fully explored in LTPs because a ternary pattern is split into a pair of binary patterns. In this paper, to fully explore the distinctiveness in the local pattern, the feature extraction process is formulated as an integer decomposition problem, which is a generalized version of the Bachet de Meziriac weight problem (BMWP). Following this generalization, a local $n$ -ary pattern (LNP) is proposed, for which the LBP is a special case parametrized under $n=2$ . The LTP is not a special case of the LNP. Both LBP and LTP are used as benchmark methods to evaluate LNPs performance due to their well-recognized success. In addition, a rotation-invariant and uniform LNP is also proposed and compared with a rotation-invariant and uniform LBP. The proposed LNP achieves significantly improved texture classification accuracy compared with the LBP and also demonstrates considerable improvement over the LTP. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a method that is able to detect fires by analyzing videos acquired by surveillance cameras. Two main novelties have been introduced. First, complementary information, based on color, shape variation, and motion analysis, is combined by a multiexpert system. The main advantage deriving from this approach lies in the fact that the overall performance of the system significantly increases with a relatively small effort made by the designer. Second, a novel descriptor based on a bag-of-words approach has been proposed for representing motion. The proposed method has been tested on a very large dataset of fire videos acquired both in real environments and from the web. The obtained results confirm a consistent reduction in the number of false positives, without paying in terms of accuracy or renouncing the possibility to run the system on embedded platforms. [ABSTRACT FROM PUBLISHER]

We present a framework for multitarget detection and tracking that infers candidate target locations in videos containing a high density of homogeneous targets. We propose a gradient-climbing technique and an isocontor slicing approach for intensity maps to localize targets. The former uses Markov chain Monte Carlo to iteratively fit a shape model onto the target locations, whereas the latter uses the intensity values at different levels to find consistent object shapes. We generate trajectories by recursively associating detections with a hierarchical graph-based tracker on temporal windows. The solution to the graph is obtained with a greedy algorithm that accounts for false-positive associations. The edges of the graph are weighted with a likelihood function based on location information. We evaluate the performance of the proposed framework on challenging datasets containing videos with high density of targets and compare it with six alternative trackers. [ABSTRACT FROM AUTHOR]

In this paper, a 2-D noncausal Markov model is proposed for passive digital image-splicing detection. Different from the traditional Markov model, the proposed approach models an image as a 2-D noncausal signal and captures the underlying dependencies between the current node and its neighbors. The model parameters are treated as the discriminative features to differentiate the spliced images from the natural ones. We apply the model in the block discrete cosine transformation domain and the discrete Meyer wavelet transform domain, and the cross-domain features are treated as the final discriminative features for classification. The support vector machine which is the most popular classifier used in the image-splicing detection is exploited in our paper for classification. To evaluate the performance of the proposed method, all the experiments are conducted on public image-splicing detection evaluation data sets, and the experimental results have shown that the proposed approach outperforms some state-of-the-art methods. [ABSTRACT FROM PUBLISHER]

Homography estimation is a fundamental problem in the field of computer vision. For estimating the homography between two images, one of the key issues is to match keypoints in the reference image to the keypoints in the moving image. To match keypoints in real time, a binary image descriptor, due to its low matching and storage costs, emerges as a more and more popular tool. Upon achieving the low costs, the binary descriptor sacrifices the discriminative power of using floating points. In this paper, we present BB-Homography, a new approach that fuses fast binary descriptor matching and bipartite graph for homography estimation. Starting with binary descriptor matching, BB-Homography uses bipartite graph matching (GM) algorithm to refine the matching results, which are finally passed over to estimate homography. On realizing the correlation between keypoint correspondence and homography estimation, BB-Homography iteratively performs the GM and the homography estimation such that they can refine each other at each iteration. In particular, based on spectral graph, a fast bipartite GM algorithm is developed for lowering the time cost of BB-Homography. BB-Homography is extensively evaluated on both public benchmarks and live-captured video streams that consistently shows that BB-Homography outperforms conventional methods for homography estimation. [ABSTRACT FROM PUBLISHER]

Every year, a large number of wildfires all over the world burn forested lands, causing adverse ecological, economic, and social impacts. Beyond taking precautionary measures, early warning and immediate response are the only ways to avoid great losses. To this end, in this paper we propose a computer vision approach for fire-flame detection to be used by an early-warning fire monitoring system. Initially, candidate fire regions in a frame are defined using background subtraction and color analysis based on a nonparametric model. Subsequently, the fire behavior is modeled by employing various spatio-temporal features, such as color probability, flickering, spatial, and spatio-temporal energy, while dynamic texture analysis is applied in each candidate region using linear dynamical systems and a bag-of-systems approach. To increase the robustness of the algorithm, the spatio-temporal consistency energy of each candidate fire region is estimated by exploiting prior knowledge about the possible existence of fire in neighboring blocks from the current and previous video frames. As a final step, a two-class support vector machine classifier is used to classify the candidate regions. Experimental results have shown that the proposed method outperforms existing state-of-the-art algorithms. [ABSTRACT FROM PUBLISHER]

A very-large-scale integration capable of extracting motion features from moving images in real time has been developed employing row-parallel and pixel-parallel architectures based on the digital pixel sensor technology. Directional edge filtering of input images is carried out in row-parallel processing to minimize the chip real estate. To achieve a real-time response of the system, a fully pixel-parallel architecture has been explored in adaptive binarization of filtered images for essential feature extraction as well as in their temporal integration and derivative operations. As a result, self-speed-adaptive motion feature extraction has been established. The chip was designed and fabricated in a 65-nm CMOS technology and used to build an object detection system. Motion-sensitive target image localization was demonstrated as an illustrative example. [ABSTRACT FROM AUTHOR]

Networks of smart cameras share large amounts of data to accomplish tasks such as reidentification. We propose a feature-selection method that minimizes the data needed to represent the appearance of objects by learning the most appropriate feature set for the task at hand (person reidentification). The computational cost for feature extraction and the cost for storing the feature descriptor are considered jointly with feature performance to select cost-effective good features. This selection allows us to improve intercamera reidentification while reducing the bandwidth that is necessary to share data across the camera network. We also rank the selected features in the order of effectiveness for the task to enable a further reduction of the feature set by dropping the least effective features when application constraints require this adaptation. We compare the proposed approach with state-of-the-art methods on the iLIDS and VIPeR datasets and show that the proposed approach considerably reduces network traffic due to intercamera feature sharing while keeping the reidentification performance at an equivalent or better level compared with the state of the art. [ABSTRACT FROM AUTHOR]

This paper introduces a high-speed all-hardware scale-invariant feature transform (SIFT) architecture with parallel and pipeline technology for real-time extraction of image features. The task-level parallel and pipeline structure are exploited between the hardware blocks, and the data-level parallel and pipeline architecture are exploited inside each block. Two identical random access memories are adopted with ping-pong operation to execute the key point detection module and the descriptor generation module in task-level parallelism. With speeding up the key point detection module of SIFT, the descriptor generation module has become the bottleneck of the system's performance; therefore, this paper proposes an optimized descriptor generation algorithm. A novel window-dividing method is proposed with square subregions arranged in 16 directions, and the descriptors are generated by reordering the histogram instead of window rotation. Therefore, the main orientation detection block and descriptor generation block run in parallel instead of interactively. With the optimized algorithm cooperating with pipeline structure inside each block, we not only improve the parallelism of the algorithm, but also avoid floating data calculation to save hardware consumption. Thus, the descriptor generation module leads the speed almost 15 times faster than a recent solution. The proposed system was implemented on field programmable gate array and the overall time to extract SIFT features for an image having \(512\times 512\) pixels is only 6.55 ms (sufficient for real-time applications), and the number of feature points can reach up to 2900. [ABSTRACT FROM PUBLISHER]

Detecting and matching image features is a fundamental task in video analytics and computer vision systems. It establishes the correspondences between two images taken at different time instants or from different viewpoints. However, its large computational complexity has been a challenge to most embedded systems. This paper proposes a new FPGA-based embedded system architecture for feature detection and matching. It consists of scale-invariant feature transform (SIFT) feature detection, as well as binary robust independent elementary features (BRIEF) feature description and matching. It is able to establish accurate correspondences between consecutive frames for 720-p (1280x720) video. It optimizes the FPGA architecture for the SIFT feature detection to reduce the utilization of FPGA resources. Moreover, it implements the BRIEF feature description and matching on FPGA. Due to these contributions, the proposed system achieves feature detection and matching at 60 frame/s for 720-p video. Its processing speed can meet and even exceed the demand of most real-life real-time video analytics applications. Extensive experiments have demonstrated its efficiency and effectiveness. [ABSTRACT FROM PUBLISHER]

Saliency detection is widely used to extract regions of interest in images for various image processing applications. Recently, many saliency detection models have been proposed for video in uncompressed (pixel) domain. However, video over Internet is always stored in compressed domains, such as MPEG2, H.264, and MPEG4 Visual. In this paper, we propose a novel video saliency detection model based on feature contrast in compressed domain. Four types of features including luminance, color, texture, and motion are extracted from the discrete cosine transform coefficients and motion vectors in video bitstream. The static saliency map of unpredicted frames (I frames) is calculated on the basis of luminance, color, and texture features, while the motion saliency map of predicted frames (P and B frames) is computed by motion feature. A new fusion method is designed to combine the static saliency and motion saliency maps to get the final saliency map for each video frame. Due to the directly derived features in compressed domain, the proposed model can predict the salient regions efficiently for video frames. Experimental results on a public database show superior performance of the proposed video saliency detection model in compressed domain. [ABSTRACT FROM AUTHOR]

An area efficient row-parallel architecture is proposed for the real-time implementation of bivariate algebraic integer (AI) encoded 2-D discrete cosine transform (DCT) for image and video processing. The proposed architecture computes 8\,\times\,8 2-D DCT transform based on the Arai DCT algorithm. An improved fast algorithm for AI-based 1-D DCT computation is proposed along with a single channel 2-D DCT architecture. The design improves on the four-channel AI DCT architecture that was published recently by reducing the number of integer channels to one and the number of eight-point 1-D DCT cores from five down to two. The architecture offers exact computation of 8\,\times\,8 blocks of the 2-D DCT coefficients up to the FRS, which converts the coefficients from the AI representation to fixed-point format using the method of expansion factors. Prototype circuits corresponding to FRS blocks based on two expansion factors are realized, tested, and verified on FPGA-chip, using a Xilinx Virtex-6 XC6VLX240T device. Post place-and-route results show a 20% reduction in terms of area compared to the 2-D DCT architecture requiring five 1-D AI cores. The area-time and area\-time^2 complexity metrics are also reduced by 23% and 22% respectively for designs with eight-bit input word length. The digital realizations are simulated up to place and route for ASICs using 45 nm CMOS standard cells. The maximum estimated clock rate is 951 MHz for the CMOS realizations indicating 7.608\cdot 10^9~pixels/s and a 8\,\times\,8 block rate of 118.875 MHz. [ABSTRACT FROM AUTHOR]

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and to preserve the spatial correlation of MVs. Furthermore, the proposed method can detect occluded regions, where a continuously tracked feature does not have a corresponding point in the current frame. If an occluded region is detected, the intermediate frame is generated from either the previous frame or the current frame. In the non-occluded region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results showed that the proposed temporal frame interpolation (TFI) method can improve the visual quality compared with conventional TFI methods, both objectively and subjectively. [ABSTRACT FROM AUTHOR]

We introduce a new objective assessment method for visual discomfort of stereoscopic images that makes effective use of the human visual attention model. The proposed method takes into account visual importance regions that play an important role in determining the overall degree of visual discomfort of a stereoscopic image. After obtaining a saliency-based visual importance map for an image, perceptually significant disparity features are extracted to predict the overall degree of visual discomfort. Experimental results show that the proposed method can achieve significantly higher prediction accuracy than the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Image registration is a challenging task, with applications in surveillance, motion estimation, and fusion systems. Due to the diversity of sensors, local distortions and large image size, satellite images are often difficult to accurately register. In the literature, local descriptor-based processing techniques, such as scale-invariant feature transforms (SIFTs), have been applied to register satellite images, which provide robust features. However, these techniques suffer from a high-computational cost, lack of features, and low-distribution quality, which affect the registration accuracy. In this paper, we develop an algorithm to register satellite images based on adaptive block processing to increase the number of features and to improve the distribution quality. In addition, outlier removal using statistical masks are associated with classical random sample consensus (RANSAC); a subsequent comparative analysis demonstrates the accuracy of the proposed method. Typically, a classical SIFT prevents its wide application in recent remote sensing, although this is no longer the case with the proposed adaptive block processing method. [ABSTRACT FROM PUBLISHER]

We propose a facial trait code (FTC) to encode human facial images, and apply it to face recognition. Extracted from an exhaustive set of local patches cropped from a large stack of faces, the facial traits and the associated trait patterns can accurately capture the appearance of a given face. The extraction has two phases. The first phase is composed of clustering and boosting upon a training set of faces with neutral expression, even illumination, and frontal pose. The second phase focuses on the extraction of the facial trait patterns from the set of faces with variations in expression, illumination, and poses. To apply the FTC to face recognition, two types of codewords, hard and probabilistic, with different metrics for characterizing the facial trait patterns are proposed. The hard codeword offers a concise representation of a face, while the probabilistic codeword enables matching with better accuracy. Our experiments compare the proposed FTC to other algorithms on several public datasets, all showing promising results. [ABSTRACT FROM PUBLISHER]

Subspace learning is an effective and widely used image feature extraction and classification technique. However, for the large-scale image recognition issue in real-world applications, many subspace learning methods often suffer from large computational burden. In order to reduce the computational time and improve the recognition performance of subspace learning technique under this situation, we introduce the idea of parallel computing which can reduce the time complexity by splitting the original task into several subtasks. We develop a parallel subspace learning framework. In this framework, we first divide the sample set into several subsets by designing two random data division strategies that are equal data division and unequal data division. These two strategies correspond to equal and unequal computational abilities of nodes under parallel computing environment. Next, we calculate projection vectors from each subset in parallel. The graph embedding technique is employed to provide a general formulation for parallel feature extraction. After combining the extracted features from all nodes, we present a unified criterion to select most distinctive features for classification. Under the developed framework, we separately propose supervised and unsupervised parallel subspace learning approaches, which are called parallel linear discriminant analysis (PLDA) and parallel locality preserving projection (PLPP). PLDA selects features with the largest Fisher scores by estimating the weighted and unweighted sample scatter, while PLPP selects features with the smallest Laplacian scores by constructing a whole affinity matrix. Theoretically, we analyze the time complexities of proposed approaches and provide the fundamental supports for applying random division strategies. In the experiments, we establish two real parallel computing environments and employ four public image and video databases as the test data. Experimental results demonstrate that the proposed approaches outperform several related supervised and unsupervised subspace learning methods, and significantly reduce the computational time. [ABSTRACT FROM PUBLISHER]

In this paper, a region-based active contour model (ACM) with local information using watershed segmentation is proposed for lips contour detection. Compared to the ACM with global energy terms, the proposed system provides a more precise lips contour convergence under the circumstances where the lips are difficult to distinguish using global statistics. Furthermore, since the ACM is sensitive to the initial contour position, a modified H\infty based on Lyapunov stability theory is proposed to provide better tracking of the subsequent lips feature points as the ACM initialization. The integration of the proposed ACM and modified H\infty has revealed an improvement of the overall lips contour detection. [ABSTRACT FROM AUTHOR]

Feature extraction is an essential part in applications that require computer vision to recognize objects in an image processed. To extract the features robustly, feature extraction algorithms are often very demanding in computation so that the performance achieved by pure software is far from real-time. Among those feature extraction algorithms, scale-invariant feature transform (SIFT) has gained a lot of popularity recently. In this paper, we propose an all-hardware SIFT accelerator—the fastest of its kind to our knowledge. It consists of two interactive hardware components, one for key point identification, and the other for feature descriptor generation. We successfully developed a segment buffer scheme that could not only feed data to the computing modules in a data-streaming manner, but also reduce about 50% memory requirement than a previous work. With a parallel architecture incorporating a three-stage pipeline, the processing time of the key point identification is only 3.4 ms for one video graphics array (VGA) image. Taking also into account the feature descriptor generation part, the overall SIFT processing time for a VGA image can be kept within 33 ms (to support real-time operation) when the number of feature points to be extracted is fewer than 890. [ABSTRACT FROM AUTHOR]

Image feature extraction and similarity measure in feature space are active research topics. They are basic components in a content-based image retrieval (CBIR) system. In this letter, we present a new statistical model-based image feature extraction method in the wavelet domain and a novel Kullback divergence-based similarity measure. First, a Gaussian mixture model (GMM) and a more systematic generalized Gaussian mixture model (GGMM) are employed to describe the statistical characteristics of the wavelet coefficients and the model parameters are employed to construct a compact image feature space. A nontrivial expectation-maximization (EM) algorithm for the GGMM model is derived. Subsequently, a new Kullback divergence-based similarity measure with lowcomputation cost is derived and analyzed. The Brodatz texture image database and some other image databases are used to evaluate the retrieval performance based on the presented new methods. Experimental results indicate that the GMM and the GGMM-based image texture features are very effective in representing multiscale image characteristics and that the new methods outperforms other conventional wavelet-based methods in retrieval performance with a comparable level of computational complexity. It is also demonstrated that for image features extracted by the new statistical models, the similarity measure based on Kullback divergence is more effective than conventional similarity measures. [ABSTRACT FROM AUTHOR]

Region-of-interest (ROI) determination is very important for video processing and it is desirable to find a simple method to identify the ROl. Along this direction, this paper investigates a user attention model based on visual rhythm analysis for automatic determination of ROl in a video. The visual rhythm, which is an abstraction of a video, is a thumbnail version of a video by a 2-D image that captures the temporal information of a video sequence. Four sampling lines, including diagonal, anti- diagonal, vertical, and horizontal lines, are employed to obtain four visual rhythm maps in order to analyze the location of the ROl from video data. Via the variation on visual rhythms, object and camera motions can be efficiently distinguished. As for hardware design consideration, the proposed scheme can accurately extract ROl with very low computational complexity for real-time applications. The promising results from the experiments demonstrate that the moving object is effectively and efficiently extracted. Finally, we present a way to use flexible macroblock ordering in combination with ROl determination as a preprocessing step for H.264/AVC video coding, and experimental results show the quality of ROl regions is significantly enhanced. [ABSTRACT FROM AUTHOR]

This paper describes a unified framework for object mining system for videos, which combines shot segmentation, clustering, retrieval and object mining using a single set of detected local invariant regions. The local invariant regions are tracked throughout a shot and stable tracks are extracted. The conventional key frame method is replaced with these stable tracks of local regions to characterize different shots. A grouping technique is introduced to combine the stable tracks into meaningful object clusters. These clusters are used to mine similar objects. Compared to other object mining systems, our approach mines more instances of similar objects in different shots. The proposed framework is applied to full length feature films and the results are compared with state of the art methods. [ABSTRACT FROM AUTHOR]

The motion compensated temporal filtering (MCTF) technique, which is extensively used in 3-D wavelet video coding schemes nowadays, leads to signal coupling among various spatial subbands because motion alignment is introduced in the temporal filtering. Using all spatial subbands as a reference enables MCTF to fully take advantage of temporal correlation across frames but inevitably brings drifting problem in supporting spatial scalability. This paper first analyzes the signal coupling phenomenon and then proposes a quantitative model to describe signal propagation across spatial subbands during the MCTF process. The signal propagation is modeled for a single MC step based on the shifting effect of wavelet synthesis filters and then it is extended to multi- level MCTF. This model is called subband coupling aware signal propagation (SCASP) model in this paper. Based on the model, we further propose a subband coupling aware rate allocation scheme as one possible solution to the above dilemma in supporting spatial scalability. To find the optimal rate allocation among all subbands for a specified reconstruction resolution, the SCASP model is used to approximate the reconstruction process and derive the synthesis gain of each subband with regard to that reconstruction. Experimental results have fully demonstrated the advantages of our proposed rate allocation scheme in improving both objective and subjective qualities of reconstructed low-resolution video, especially at middle bit rates and high bit rates. [ABSTRACT FROM AUTHOR]

This work presents a new class of algorithms for super-resolution (SR) of image sequences. This class of algorithms estimates simultaneously all frames of a sequence by employing an iterative minimization of a regularized cost function. Similarly to other SR techniques, the proposed approach exploits the correlation among the frames of the sequence. This correlated information helps to improve the resolution of the captured images. By employing the motion information only in the prior term of the cost function, the proposed method achieves a better fidelity and more robust performance, when compared to other methods. This original approach provides a fidelity equivalent to the obtained by the simultaneous methods, while achieving a lower computational complexity. The isolation of the motion equations in the prior term allows an increased control over the motion errors, and as a consequence, provides an improved robustness. The performance of the proposed method is discussed and compared with other methods in the literature. In the comparative experiments, it is considered both the minimization over the Euclidean norm, which is employed to achieve low computational complexity, and the minimization over the Huber norm or ℓ1 norm, which is used to produce images with sharp edges and higher robustness to large motion errors. [ABSTRACT FROM AUTHOR]