Search

Images captured under low-light environments typically have poor visibility, affecting many advanced computer vision tasks. In recent years, there have been some low-light image enhancement models based on deep learning, but they have not been able to effectively mine the deep multiscale features in the image, resulting in poor generalization performance and instability of the model. The disadvantages are mainly reflected in the color distortion, color unsaturation and artifacts. Current methods unable to adjust the exposure effectively, resulting in uneven exposure or partial overexposure. To address these issues, we propose an end-to-end low-light image enhancement model, which is called multiscale low-light image enhancement network with illumination constraint (MLLEN-IC), to achieve preferable generalization ability and stable performance. On the one hand, we use the squeeze-and-excitation-Res2Net block (SE-Res2block) as a base unit to enhance the model’s ability by extracting deep multiscale features. On the other hand, to make the model more adaptable in low-light image enhancement tasks, we calculate the illumination constraint by the low-light itself to prevent overexposure, uneven exposure, and unsaturated colors. Extensive experiments are conducted to demonstrate MLLEN-IC not only adjusts light levels, but also has a more natural visual effect, and avoids problems such as color distortion, artifacts, and uneven exposure. In particular, MLLEN-IC has pretty generalization and stability performance. The source code and supplementary are available at https://github.com/CCECfgd/MLLEN-IC. [ABSTRACT FROM AUTHOR]

Optical flow estimation is a fundamental task in computer vision and image processing. Due to the difficulty in obtaining the ground truth of flow field, unsupervised learning approaches attract more and more research interests in recent years. However, despite of their good generalization capability, unsupervised optical flow methods suffer in the scenarios with large displacement, small objects, and occlusions. In this work, we propose a novel optical flow network based on decoder with multi-scale kernels. Different from previous U-Net like or pyramidal methods, we design our network based on RAFT architecture that with a 4D correlation layer and recurrent decoder. More importantly, we incorporate three novel ideas with regard to the input, information processing and output of the update units improve the performance. Firstly, we utilize various motion-related information as input to the update units. Secondly, we propose a module of multi-scale update unit. Thirdly, for the final flow up-sampling procedure, we propose an image-guided up-sampling loss to guide the learning of up-sampling masks. Our model is trained by the occlusion-aware photometric loss, edge-aware smoothness loss, self-supervised loss, and image-guided up-sampling loss. Experimental results demonstrate that our model achieves the state-of-the-art performance on both Sintel and KITTI and outperforms other unsupervised optical flow methods remarkably. [ABSTRACT FROM AUTHOR]

Joint photographic experts group (JPEG) compression is widely used in image processing and computer vision. Detecting double compressed JPEG images is a common problem in forensics and detecting compressed images with the same quantization matrix remains a challenging task. However, most existing methods were designed for detection in grayscale images and cannot fully use the unique characteristics of color images (such as the relationship between channels and color information). In addition, the performance of existing methods is unsatisfactory for low JPEG quality factors and in cross detection experiments. To solve these problems, we analyze the stability of a color image to obtain the convergence error and transposition error. According to the convergence characteristics of color JPEG images, the continuous compression by the same quantization matrix can make the JPEG image tend to be stable. The final stable state and the convergence process are determined by the number of compressions of the original image. Thus, continuously compressed JPEG images can be regarded as a continuous frame to obtain the convergence error. As the color image converges, its ability to resist interference decreases. To reflect the changes in anti-interference ability, the transposition operation is used to disturb the color JPEG image to obtain the transposition error. In addition, quaternion mapping is used to retain the relationship between continuously compressed JPEG images and enlarge the influence caused by transposition operation. In our experiments on several image databases, the proposed method outperforms existing methods in different settings. [ABSTRACT FROM AUTHOR]

It is challenging to extend the support region of state-of-the-art local edge-preserving filtering approaches to the entire input image on account of huge memory cost and heavy computational burden. In this paper, we propose an O(N) time recursive non-local edge-aware filter. A novel graph and a linear time algorithm are presented to effectively propagate information across the entire image. In this graph, information is propagated along all directions to avoid visual artifacts. Both the intensity similarity and the spatial affinity are utilized to estimate the similarity of neighboring pixels, and the distance of any two pixels is the length of the path between these two pixels on a binary tree. Specifically, the input image is filtered at four directions, namely left-to-right, right-to-left, top-to-bottom and bottom-to-top. In each pass, the un-normalized output and the normalization constant of the root node are computed recursively from leaf nodes to the root node on a binary tree in linear time. The filtering output is the average of the outputs for these four directions. A comparison with other non-local edge-aware filters is presented to show the advantages of our approach. Extensive experiments demonstrate that our recursive non-local edge-preserving filter is effective in a variety of computer vision and image processing applications, including edge-aware filtering, detail enhancement, stylization, and stereo matching. [ABSTRACT FROM AUTHOR]

Being able to predict the salient object is of fundamental importance in image processing and computer vision. With numerous approaches proposed for automatic image and video salient object detection, much less work has been dedicated to detecting and segmenting salient objects from light fields. In this article, based on the intrinsic characteristics of light fields, we carefully explore the complementary coherence among multiple cues including spatial, edge and depth information, and elaborately design a multi-task collaborative network for light field salient object detection. More specifically, the correlation mechanisms among edge detection, depth inference and salient object detection are carefully investigated to facilitate the representative saliency features. We first model the coherence among low-level features and heuristic semantic priors, as well as the edge information. Subsequently, the depth-oriented saliency features are derived from the geometry of light fields, in which the 3D convolution operation is leveraged with powerful representation capability to model the disparity correlations among multiple viewpoint images. Finally, a feature-enhanced salient object generator is developed to integrate these complementary saliency features, leading to the final salient object predictions for light fields. Quantitative and qualitative experiments demonstrate the superiority of our proposed model against the state-of-the-art methods over the public light field salient object detection datasets. [ABSTRACT FROM AUTHOR]

Color and color differences are critical aspects in many image processing and computer vision applications. A paradigmatic example is object segmentation, where color distances can greatly influence the performance of the algorithms. Metrics for color difference have been proposed in the literature, including the definition of standards such as CIEDE2000, which quantifies the change in visual perception of two given colors. This standard has been recommended for industrial computer vision applications, but the benefits of its application have been impaired by the complexity of the formula. This paper proposes a new strategy that improves the usability of the CIEDE2000 metric when a maximum acceptable distance can be imposed. We argue that, for applications where a maximum value, above which colors are considered to be different, can be established, then it is possible to reduce the amount of calculations of the metric, by preemptively analyzing the color features. This methodology encompasses the benefits of the metric while overcoming its computational limitations, thus broadening the range of applications of CIEDE2000 in both the computer vision algorithms and computational resource requirements. [ABSTRACT FROM AUTHOR]

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study. [ABSTRACT FROM AUTHOR]

Human action recognition (HAR) is an important field of research that intercepts with areas such as image processing, computer vision, and the design of fast algorithms, among others. HAR has several important applications including healthcare monitoring, security and surveillance, assisted living, smart homes, and video search and indexing. Despite recent developments in the field, major challenges remain. For instance, HAR is computationally expensive. Tasks such as video preprocessing, feature extraction, feature quantization, and feature classification require the execution of millions of arithmetic operations for a video sequence lasting a few seconds. To address these problems, we propose a heterogeneous approach that is based on an extensive algorithmic and experimental analysis of the histogram of gradients application. We divide the application into four stages and evaluate each on the CPU, GPU, and FPGA platforms. Our heterogeneous design combines the strengths of both the FPGA and GPU platforms, and achieves a $1.3X$ speedup compared with a state-of-the-art GPU while being $1.5X$ more energy efficient than other homogeneous solutions, including FPGA-based designs. Moreover, our heterogeneous HAR design using fixed-point arithmetic has comparable accuracy to those of HAR algorithms using single precision floating point arithmetic. [ABSTRACT FROM AUTHOR]

Ill-posed problems are very common in many image processing and computer vision tasks. To solve such problems, a regularization must be imposed. In this paper, we argue why mean curvature is a good regularization for these tasks. From a geometry point of view, we show that minimizing mean curvature is to assume that the ground truth is a piece-wise minimal surface. From a statistics point of view, we show that the mean curvature from natural images is sparse. From an optimization point of view, we show that the gradient of mean curvature regularization can be numerically approximated by very simple filters. This fact significantly simplifies the optimization procedure. Thus, the link between these filters and the gradient of mean curvature regularization is established in this paper. In summary, mean curvature is a proper regularization for various ill-posed problems in image processing and computer vision. [ABSTRACT FROM AUTHOR]

We introduce a novel method to automatically adjust camera exposure for image processing and computer vision applications on mobile robot platforms. Because most image processing algorithms rely heavily on low-level image features that are based mainly on local gradient information, we consider that gradient quantity can determine the proper exposure level, allowing a camera to capture the important image features in a manner robust to illumination conditions. We then extend this concept to a multi-camera system and present a new control algorithm to achieve both brightness consistency between adjacent cameras and a proper exposure level for each camera. We implement our prototype system with off-the-shelf machine-vision cameras and demonstrate the effectiveness of the proposed algorithms on practical applications, including pedestrian detection, visual odometry, surround-view imaging, panoramic imaging, and stereo matching. [ABSTRACT FROM AUTHOR]

Low-light image enhancement algorithms can improve the visual quality of low-light images and support the extraction of valuable information for some computer vision techniques. However, existing techniques inevitably introduce color and lightness distortions when enhancing the images. To lower the distortions, we propose a novel enhancement framework using the response characteristics of cameras. First, we discuss how to determine a reasonable camera response model and its parameters. Then, we use the illumination estimation techniques to estimate the exposure ratio for each pixel. Finally, the selected camera response model is used to adjust each pixel to the desired exposure according to the estimated exposure ratio map. Experiments show that our method can obtain enhancement results with fewer color and lightness distortions compared with the several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Despite recent advances made in salient object detection using graph theory, the approach still suffers from accuracy problems when the image is characterized by a complex structure, either in the foreground or background, causing erroneous saliency segmentation. This fundamental challenge is mainly attributed to the fact that most existing graph-based methods take only the adjacently spatial consistency among graph nodes into consideration. In this paper, we tackle this issue from a coarse-to-fine perspective and propose a two-stage-graphs approach for salient object detection, in which two graphs having the same nodes but different edges are employed. Specifically, a weighted joint robust sparse representation model, rather than the commonly used manifold ranking model, helps to compute the saliency value of each node in the first-stage graph, thereby providing a saliency map at the coarse level. In the second-stage graph, along with the adjacently spatial consistency, a new regionally spatial consistency among graph nodes is considered in order to refine the coarse saliency map, assuring uniform saliency assignment even in complex scenes. Particularly, the second stage is generic enough to be integrated in existing salient object detectors, enabling improved performance. Experimental results on benchmark data sets validate the effectiveness and superiority of the proposed scheme over related state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Despite the recent success of 3D action recognition using depth sensor, most existing works target how to improve the action recognition performance, rather than understanding how different types of actions are performed. In this paper, we propose to discover discriminative spatio-temporal patterns for 3D action recognition. Discovering these patterns can not only help to improve the action recognition performance but also help us to understand and differentiate between the action category. Our proposed method takes the spatio-temporal structure of 3D action into consideration and can discover essential spatio-temporal patterns that play key roles in action recognition. Instead of relying on an end-to-end network to learn the 3D action representation and perform classification, we simply present each 3D action as a series of temporal stages composed by 3D poses. Then, we rely on nearest neighbor matching and bilinear classifiers to simultaneously identify both critical temporal stages and spatial joints for each action class. Despite using raw action representation and a linear classifier, experiments on five benchmark data sets show that the proposed spatio-temporal naïve Bayes mutual information maximization can achieve a competitive performance compared with the state-of-the-art methods that use sophisticated end-to-end learning, and has the advantage of finding discriminative spatio-temporal action patterns. [ABSTRACT FROM AUTHOR]

Although encouraging results have been obtained in human pose estimation in recent years, the performance may degrade dramatically when the image quality differs between training and testing data sets. This paper addresses problems in cross-image-quality human pose estimation. To achieve this, we follow an unsupervised domain adaptation approach, in which labels in the target domain are unavailable. Unlike existing unsupervised domain adaptation methods that find label information from unlabeled data, the target pose information (label) is instead generated by synthesizing body parts with similar image-quality of the target domain. A translative dictionary is learned to associate the source and target domains, and a cross-quality adaptation model is developed to refine the source pose estimator using the synthesized target body parts. We perform cross-quality experiments on three data sets with different image quality by using two state-of-the-art pose estimators, and compare the proposed method with five unsupervised domain adaptation methods. Our experimental results show that the proposed method outperforms not only the source pose estimators, but also other unsupervised domain adaptation methods. [ABSTRACT FROM AUTHOR]

Objectness has recently become a standard step in many computer vision tasks. Among various techniques, those based on hierarchical image segmentation play a fundamental role for developments in new data modalities. In this paper, we address the problem of objectness in RGB-D images and propose a novel and effective approach, namely, hierarchical image segmentation ensemble (HISE). Different from existing image segmentation based methods that generate object segments or proposals largely by heuristics or empirical rules, HISE learns superpixel mergings with a hierarchical tree-structured ensemble, where individual merging models of the ensemble are formed by traversing different paths of the tree, and where both the merging accuracy and proposal diversity are emphasized. Furthermore, we use efficient feature measurements that support easy integration of additional clues. Extensive experiments conducted on the benchmark NYU-v2 RGB-D and SUN RGB-D data sets show the competency of our proposed method. [ABSTRACT FROM AUTHOR]

Gesture recognition has raised wide attention in computer vision owing to its many applications. However, the task of video-based large-scale gesture recognition yet faces many challenges, since many gesture-irrelevant factors like the background may disturb the recognition accuracy. To better recognize gestures with large-scale videos, we propose a method based on RGB-D data in this paper, where the “RGB-D” means RGB and depth data captured simultaneously by specific devices like Kinect. To learn gesture details better, we first use an adaptive frame unification strategy to unify the frame number of inputs, and then the RGB and depth data are sent to the C3D model to extract spatiotemporal features, respectively. In order to alleviate the interference of gesture-irrelevant factors, the saliency theory is also employed to generate auxiliary data. Next the features of these data are combined to boost the performance, which can also avoid unreasonable synthetic data, since the dimension of C3D features is uniform. Finally the performances of several classifiers are tested and the best one of SVM classifier is selected to output the ultimate accuracy. Our approach achieves 52.04% and 59.43% accuracy on the validation and testing subset of the Chalearn LAP IsoGD, respectively, both of which outperform our results in the chalearn LAP Large-scale Gesture Recognition Challenge as reported in ICPR 2016. [ABSTRACT FROM AUTHOR]

We consider the person re-identification problem, assuming the availability of a sequence of images for each person, commonly referred to as video-based or multi-shot re-identification. We approach this problem from the perspective of learning discriminative distance metric functions. While existing distance metric learning methods typically employ the average feature vector as the data exemplar, this discards the inherent structure of the data. To overcome this issue, we describe the image sequence data using affine hulls. We show that directly computing the distance between the closest points on these affine hulls as in existing recognition algorithms is not sufficiently discriminative in the context of person re-identification. To this end, we incorporate affine hull data modeling into the traditional distance metric learning framework, learning discriminative feature representations directly using affine hulls. We perform extensive experiments on several publicly available data sets to show that the proposed approach improves the performance of existing metric learning algorithms irrespective of the feature space employed to perform metric learning. Furthermore, we advance the state of the art on iLIDS-VID, PRID, and SAIVT, with absolute rank-1 performance improvements of 6.0%, 11.4%, and 6.0% respectively. [ABSTRACT FROM AUTHOR]

This paper presents a robust joint discriminative appearance model-based tracking method using online random forests and mid-level feature (superpixels). To achieve superpixel-wise discriminative ability, we propose a joint appearance model that consists of two random forest-based models, i.e., the background-target discriminative model (BTM) and the distractor-target discriminative model (DTM). More specifically, the BTM effectively learns discriminative information between the target object and the background. In contrast, the DTM is used to suppress distracting superpixels, which significantly improves the tracker’s robustness and alleviates the drifting problem. A novel online random forest regression algorithm is proposed to build the two models. The BTM and DTM are linearly combined into a joint model to compute a confidence map. Tracking results are estimated using the confidence map, in which the position and scale of the target are estimated orderly. Furthermore, we design a model updating strategy to adapt the appearance changes over time by discarding degraded trees of the BTM and DTM and initializing new trees as replacements. We test the proposed tracking method on two large tracking benchmarks, the CVPR2013 tracking benchmark and VOT2014 tracking challenge. Experimental results show that the tracker runs at real-time speed and achieves favorable tracking performance compared with the state-of-the-art methods. The results also suggest that the DTM improves tracking performance significantly and plays an important role in robust tracking. [ABSTRACT FROM AUTHOR]

This paper proposes an efficient real-time background subtraction algorithm, which is essential in many computer vision applications. Initially, histograms of the intensity values for each channel of a pixel position in a set of training framesare constructed. A background model, histogram min-max bucket, is constructed from the minimum and maximum values of contiguous non-zero frequencies of the temporal intensity histogram. A novel feature of this algorithm is the use of a single sliding window to update the system adaptively, capturing the proper background even under sudden and/or gradual illumination changes in the scene. We incorporate both local and global information with the help of the current and the previously visited pixel values for binary classification of a pixel into foreground and background. This algorithm is compared with several state-of-the-art techniques and experimental studies show that the proposed method outperforms all these methods in terms of accurate binary classification. [ABSTRACT FROM AUTHOR]

We present a new statistical approach combining temporal and spatial information for robust online background subtraction (BS) in videos. Temporal information is modeled by coupling finite mixtures of generalized Gaussian distributions with foreground/background co-occurrence analysis. Spatial information is modeled by combining multiscale inter-frame correlation analysis and histogram matching. We propose an online algorithm that efficiently fuses both information to cope with several BS challenges, such as cast shadows, illumination changes, and various complex background dynamics. In addition, global video information is used through a displacement measuring technique to deal with pan-tilt-zoom camera effects. Experiments with comparison with recent state-of-the-art methods have been conducted on standard data sets. Obtained results have shown that our approach surpasses several state-of-the-art methods on the aforementioned challenges while maintaining comparable computational time. [ABSTRACT FROM AUTHOR]

Object recognition is widely used in vision computing for various applications. Traditional CPU and application specific integrated circuit for vision computing cannot provide high performance and enough flexibility, which limit the use of vision systems. In this paper, a hierarchical and parallel pipelined heterogeneous chip for object recognition is proposed to achieve high flexibility, high performance, and area efficiency. In addition, a reformulation of 3D position estimation is proposed. The method uses single precision to achieve the short computing time and accuracy requirement. The hardware resource is small. Application-specific components, such as connected component information extractor and information extraction accelerator, are designed for high performance. Reconfiguration processors and application-specific instruction set processor are introduced to improve flexibility. These components are connected to hierarchical parallel buses. The chip is fabricated in 180-nm CMOS technology and occupies 72.25 mm2 with 1.09M bits on-chip memory. It delivers 204 GOPS + 665M FLOPS operations. The results show that this hierarchical and parallel pipelined heterogeneous chip is suitable for embedded vision systems. [ABSTRACT FROM AUTHOR]

Objective image quality metrics (IQMs) potentially benefit from the addition of visual saliency. However, challenges to optimizing the performance of saliency-based IQMs remain. A previous eye-tracking study has shown that gaze is concentrated in fewer places in images with highly salient features than in images lacking salient features. From this, it can be inferred that the former are more likely to benefit from adding a saliency term to an IQM. To understand whether these ideas still hold when using computational saliency instead of eye-tracking data, we first conducted a statistical evaluation using 15 state-of-the-art saliency models and 10 well-known IQMs. We then used the results to devise an algorithm, which adaptively incorporates saliency in IQMs for natural scenes, based on saliency dispersion. Experimental results demonstrate that this can give significant improvements. [ABSTRACT FROM AUTHOR]

Background modeling constitutes the building block of many computer-vision tasks. Traditional schemes model the background as a low rank matrix with corrupted entries. These schemes operate in batch mode and do not scale well with the data size. Moreover, without enforcing spatiotemporal information in the low-rank component, and because of occlusions by foreground objects and redundancy in video data, the design of a background initialization method robust against outliers is very challenging. To overcome these limitations, this paper presents a spatiotemporal low-rank modeling method on dynamic video clips for estimating the robust background model. The proposed method encodes spatiotemporal constraints by regularizing spectral graphs. Initially, a motion-compensated binary matrix is generated using optical flow information to remove redundant data and to create a set of dynamic frames from the input video sequence. Then two graphs are constructed, one between frames for temporal consistency and the other between features for spatial consistency, to encode the local structure for continuously promoting the intrinsic behavior of the low-rank model against outliers. These two terms are then incorporated in the iterative Matrix Completion framework for improved segmentation of background. Rigorous evaluation on severely occluded and dynamic background sequences demonstrates the superior performance of the proposed method over state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Eye movements play a very significant role in human–computer interaction (HCI) as they are natural and fast, and contain important cues for human cognitive state and visual attention. Over the last two decades, many techniques have been proposed to accurately estimate the gaze. Among these, video-based remote eye trackers have attracted much interest, since they enable nonintrusive gaze estimation. To achieve high estimation accuracies for remote systems, user calibration is inevitable in order to compensate for the estimation bias caused by person-specific eye parameters. Although several explicit and implicit user calibration methods have been proposed to ease the calibration burden, the procedure is still cumbersome and needs further improvement. In this paper, we present a comprehensive analysis of regression-based user calibration techniques. We propose a novel weighted least squares regression-based user calibration method together with a real-time cross-ratio based gaze estimation framework. The proposed system enables to obtain high estimation accuracy with minimum user effort, which leads to user-friendly HCI applications. Experimental results conducted on both simulations and user experiments show that our framework achieves a significant performance improvement over the state-of-the-art user calibration methods when only a few points are available for the calibration. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a novel matching method to establish dense correspondences automatically between two images in a hierarchical superpixel-to-pixel manner. Our method first estimates dense superpixel pairings between the two images in the coarse-grained level to overcome large patch displacements and then utilizes superpixel level pairings to drive the matchings in the pixel level to obtain fine texture details. In order to compensate for the influence of color and illumination variations, we apply a regularization technique to rectify images by a color transfer function. Experimental validation on benchmark data sets demonstrates that our approach achieves better visual quality outperforming the state-of-the-art dense matching algorithms. [ABSTRACT FROM PUBLISHER]

The use of local features for image representation has become popular in recent years. Local features are often used in the bag-of-visual-words scheme. Although proven effective, this method still has two drawbacks. First, local regions from which local features are extracted are not discriminative enough for visual tasks. Hence, the combination of local features is necessary. Second, the semantic gap between visual features and human perception also hinders the performance. To address these two problems, in this paper, we propose a novel contextual exemplar classifier-based method for image representation and apply it for classification tasks. Each exemplar classifier is trained to separate one training image from the other images of different classes. We partition each image into a number of regions and use the responses of these exemplar classifiers as the image region’s representation. The contextual relationship is then modeled using mixture Dirichlet distributions. A bilayer model is used to predict image classes with L2 constraints. Experimental results on the Natural Scene, Caltech-101/256, Flower-17/102, and SUN-397 data sets show that the proposed method is able to outperform the state-of-the-art local feature-based methods for image classification. [ABSTRACT FROM PUBLISHER]

We propose an online tracking algorithm for multiple target tracking with multiple cameras. In this paper, we suggest a multiple hypothesis tracking (MHT) framework to find an unknown number of multiple tracks through the spatio-temporal association between tracklets generated from multiple cameras. In this framework, the MHT is realized online by solving the maximum weighted clique problem (MWCP) at every frame to estimate the 3D trajectories of the targets. To handle the NP-hard issue of the MWCP, we propose a novel online scheme that formulates the MWCP using feedback information from the previous frame’s result to find optimal tracks at every frame. This scheme enables the MWCP to be formulated by multiple subproblems and will significantly reduce the computation. The experiments show that the proposed algorithm performs comparably with the state-of-the-art batch algorithms, even though it adopts an online scheme. [ABSTRACT FROM PUBLISHER]

Stereo matching is a challenging problem, and high-accuracy stereo matching is still required in various computer vision applications, e.g., 3-D scanning, autonomous navigation, and 3-D reconstruction. Therefore, we present a novel image-guided stereo matching algorithm, which employs the efficient combined matching cost and multistep disparity refinement, to improve the accuracy of existing local stereo matching algorithms. Different from all the other methods, we introduce a guidance image for the whole algorithm. This filter-based guidance image is generated by extracting the enhanced information from the raw stereo image. The combined matching cost consists of the novel double-RGB gradient, the improved lightweight census transform, and the image color. This cost measurement is robust against image noise and textureless regions in computing the matching cost. Furthermore, a new systemic multistep refinement process, which includes outlier classification, four-direction propagation, leftmost propagation, and an exponential step filter, is proposed to remove the outliers in the raw disparity map. Experiments on the Middlebury benchmark demonstrate our algorithm’s superior performance that it ranks first among the 158 submitted algorithms. Moreover, the proposed method is also robust on the 30 Middlebury data sets and the real-world Karlsruhe Institute of Technology and Toyota Technological Institute benchmark. [ABSTRACT FROM AUTHOR]

Three-dimensional hand gesture recognition has attracted increasing research interests in computer vision, pattern recognition, and human-computer interaction. The emerging depth sensors greatly inspired various hand gesture recognition approaches and applications, which were severely limited in the 2D domain with conventional cameras. This paper presents a survey of some recent works on hand gesture recognition using 3D depth sensors. We first review the commercial depth sensors and public data sets that are widely used in this field. Then, we review the state-of-the-art research for 3D hand gesture recognition in four aspects: 1) 3D hand modeling; 2) static hand gesture recognition; 3) hand trajectory gesture recognition; and 4) continuous hand gesture recognition. While the emphasis is on 3D hand gesture recognition approaches, the related applications and typical systems are also briefly summarized for practitioners. [ABSTRACT FROM AUTHOR]

A resource-efficient hardware architecture for connected component analysis (CCA) of streamed video data is presented, which reduces the required hardware resources, especially for larger image widths. On-chip memory requirements increase with image width and dominate the resources of state-of-the-art CCA single-pass hardware architectures. A reduction in on-chip memory resources is essential to meet the ever increasing image sizes of high-definition (HD) and ultra-HD standards. The proposed architecture is resource efficient due to several innovations. An improved label recycling scheme detects the last pixel of an image object in the video stream only a few clock cycles after its occurrence, allowing the reuse of a label in the following image row. The coordinated application of these techniques leads to significant memory savings of more than two orders in magnitude compared with classical two-pass connected component labeling architectures. Compared with the most memory-efficient state-of-the-art single-pass CCA hardware architecture, 42% or more of on-chip memory resources are saved depending on the features extracted. Based on these savings, it is possible to realize an architecture processing video streams of larger images sizes, or to use a smaller and more energy-efficient field-programmable gate array device, or to increase the functionality of already existing image processing pipelines in reconfigurable computing and embedded systems. [ABSTRACT FROM AUTHOR]

Texture image classification is important in computer vision research. To effectively capture texture patterns, a distinctive feature such as a local binary pattern (LBP) is needed. An LBP is robust against monotonic and gray-scale variations and it computes quickly. Its robustness and speed advantage have made it popular in various texture analysis applications. However, an LBP is sensitive to noise, particularly smooth weak illumination gradients in near-uniform regions. To mitigate the effect of noise and increase distinctiveness, a local ternary pattern (LTP) is proposed. Compared with a binary coding LBP, an LTP adopts ternary coding. As a result, an LTP can better tolerate noise and is significantly more distinctive. These advantages of an LTP effectively improve its classification accuracy. However, the potential of ternary coding is not fully explored in LTPs because a ternary pattern is split into a pair of binary patterns. In this paper, to fully explore the distinctiveness in the local pattern, the feature extraction process is formulated as an integer decomposition problem, which is a generalized version of the Bachet de Meziriac weight problem (BMWP). Following this generalization, a local $n$ -ary pattern (LNP) is proposed, for which the LBP is a special case parametrized under $n=2$ . The LTP is not a special case of the LNP. Both LBP and LTP are used as benchmark methods to evaluate LNPs performance due to their well-recognized success. In addition, a rotation-invariant and uniform LNP is also proposed and compared with a rotation-invariant and uniform LBP. The proposed LNP achieves significantly improved texture classification accuracy compared with the LBP and also demonstrates considerable improvement over the LTP. [ABSTRACT FROM PUBLISHER]

The visibility of outdoor images captured in inclement weather is often degraded due to the presence of haze, fog, sandstorms, and so on. Poor visibility caused by atmospheric phenomena in turn causes failure in computer vision applications, such as outdoor object recognition systems, obstacle detection systems, video surveillance systems, and intelligent transportation systems. In order to solve this problem, visibility restoration (VR) techniques have been developed and play an important role in many computer vision applications that operate in various weather conditions. However, removing haze from a single image with a complex structure and color distortion is a difficult task for VR techniques. This paper proposes a novel VR method that uses a combination of three major modules: 1) a depth estimation (DE) module; 2) a color analysis (CA) module; and 3) a VR module. The proposed DE module takes advantage of the median filter technique and adopts our adaptive gamma correction technique. By doing so, halo effects can be avoided in images with complex structures, and effective transmission map estimation can be achieved. The proposed CA module is based on the gray world assumption and analyzes the color characteristics of the input hazy image. Subsequently, the VR module uses the adjusted transmission map and the color-correlated information to repair the color distortion in variable scenes captured during inclement weather conditions. The experimental results demonstrate that our proposed method provides superior haze removal in comparison with the previous state-of-the-art method through qualitative and quantitative evaluations of different scenes captured during various weather conditions. [ABSTRACT FROM AUTHOR]

Saliency detection has been extensively studied due to its promising contributions for various computer vision applications. However, most existing methods are easily biased toward edges or corners, which are statistically significant, but not necessarily relevant. Moreover, they often fail to find salient regions in complex scenes due to ambiguities between salient regions and highly textured backgrounds. In this paper, we present a novel unified framework for spatiotemporal saliency detection based on textural contrast. Our method is simple and robust, yet biologically plausible; thus, it can be easily extended to various applications, such as image retargeting, object segmentation, and video surveillance. Based on various datasets, we conduct comparative evaluations of 12 representative saliency detection models presented in the literature, and the results show that the proposed scheme outperforms other previously developed methods in detecting salient regions of the static and dynamic scenes. [ABSTRACT FROM AUTHOR]

Detecting and matching image features is a fundamental task in video analytics and computer vision systems. It establishes the correspondences between two images taken at different time instants or from different viewpoints. However, its large computational complexity has been a challenge to most embedded systems. This paper proposes a new FPGA-based embedded system architecture for feature detection and matching. It consists of scale-invariant feature transform (SIFT) feature detection, as well as binary robust independent elementary features (BRIEF) feature description and matching. It is able to establish accurate correspondences between consecutive frames for 720-p (1280x720) video. It optimizes the FPGA architecture for the SIFT feature detection to reduce the utilization of FPGA resources. Moreover, it implements the BRIEF feature description and matching on FPGA. Due to these contributions, the proposed system achieves feature detection and matching at 60 frame/s for 720-p video. Its processing speed can meet and even exceed the demand of most real-life real-time video analytics applications. Extensive experiments have demonstrated its efficiency and effectiveness. [ABSTRACT FROM PUBLISHER]

In real-time memory-intensive image processing and vision applications, increasing image resolution requires the use of external SDR/DDR memories. However, the arbitrary pixel access patterns used in most algorithms reduce their memory throughput as a result of increasing access latency. Efficient cache design is paramount in real-time memory-intensive applications. Its effectiveness depends on the spatial and temporal locality of data access. In image processing, the spatial locality denotes the neighboring pixels, located horizontally and vertically in 2-D. However, the conventional caches used in general processors cannot define the vertical locality. We propose a rolling cache optimized for image format and algorithms, a method to reduce the miss penalty by moving the cache horizontally and vertically, and a parallel processing architecture with interpolation, multilevel and multiple caches. To support our idea, we compare it with other types of caches and show that the average memory access time and the memory bandwidth are decreased by 28% and 74%, respectively, for a 2048\,\times\,2048 image. Its performance is greater than that of the 16-way set associative cache, but the tag memory size is a bit larger than that of the direct-mapped cache. Using two different applications, we show that the proposed architecture is applicable to a number of algorithms if data access follows an arbitrary curve or block-wise pattern, which is the usual case with image processing and vision algorithms. If an application is based on local data access in resource-limited systems, it is possible to achieve high performance with lower operational frequency using the proposed architecture. [ABSTRACT FROM PUBLISHER]

The local binary pattern (LBP) and its variants have been widely investigated in many image processing and computer vision applications due to their robust ability to capture local image structures and their computational simplicity. The existing LBPs extract local structure information by establishing a relationship between the central pixel and its adjacent pixels. However, most LBPs miss the relationship among all of the pixels in the local region. Therefore, this paper proposes a novel model to establish this relationship by introducing a support LBP. The proposed model improves the performance of the existing LBP methods and results in lower sensitivity to illumination changes and radiometric variations. Moreover, the proposed model has been successfully investigated in two applications: disparity map generation and texture classification. For disparity map generation, the proposed model reduces the root mean square (RMS) error by 23.6% (in Baby1 dataset, Middlebury), and 16.58% (in Aloe dataset, Middlebury) as compared with the standard LBP under radiometric variation conditions. Moreover, the proposed model reduces the RMS by 28.11% as compared with the standard LBP under the Gaussian noise condition in the ESATS dataset. For texture classification applications, the proposed model improves the classification results from 96.26% to 98.13% on the Outext database, from 88.03% to 91.41% on the Xu database, and from 94.00% to 96.67% on the KTH-TIPS database as compared with the completed LBP. [ABSTRACT FROM PUBLISHER]

Understanding human behaviors is a challenging problem in computer vision that has recently seen important advances. Human behavior understanding combines image and signal processing, feature extraction, machine learning, and 3-D geometry. Application scenarios range from surveillance to indexing and retrieval, from patient care to industrial safety and sports analysis. Given the broad set of techniques used in video-based behavior understanding and the fast progress in this area, in this paper we organize and survey the corresponding literature, define unambiguous key terms, and discuss links among fundamental building blocks ranging from human detection to action and interaction recognition. The advantages and the drawbacks of the methods are critically discussed, providing a comprehensive coverage of key aspects of video-based human behavior understanding, available datasets for experimentation and comparisons, and important open research issues. [ABSTRACT FROM PUBLISHER]

This paper presents a system of converting conventional monocular videos to stereoscopic ones. In the system, an input monocular video is firstly segmented into shots so as to reduce operations on similar frames. An automatic depth estimation method is proposed to compute the depth maps of the video frames utilizing three monocular depth cues—depth-from-defocus, aerial perspective, and motion. Foreground/background objects can be interactively segmented on selected key frames and their depth values can be adjusted by users. Such results are propagated from key frames to nonkey frames within each video shot. Equipped with a depth-to-disparity conversion module, the system synthesizes the counterpart (either left or right) view for stereoscopic display by warping the original frames according to their disparity maps. The quality of converted videos is evaluated by human mean opinion scores, and experiment results demonstrate that the proposed conversion method achieves encouraging performance. [ABSTRACT FROM AUTHOR]

Detecting pedestrians efficiently and accurately is a fundamental step for many computer vision applications, such as smart cars and robotics. In this paper, we introduce a pedestrian detection system to extract human objectives using an on-board monocular camera. First of all, we use an experiment to demonstrate that the orientation information is critical in human detection. Secondly, the local binary patterns-based feature, oriented LBP (OLBP), is discussed. The OLBP feature integrates pixel intensity difference with texture orientation information to capture salient object features. Thirdly, a set of edge orientation histogram (EOH) and OLBP-based intrablock and interblock features is presented to describe cell-level and block-level structure information. These multilevel features capture larger-scale structure information which is more informative for pedestrian localization. Experiments on the Institut national de recherche en informatique et en automatique (INRIA) dataset and the Caltech pedestrian detection benchmark demonstrate that the new pedestrian detection system is not only comparable to the existing pedestrian detectors, but also performs at a faster speed. [ABSTRACT FROM AUTHOR]

This paper proposes a new motion estimation (ME)-specific instruction-set processor (MESIP) with a novel search scan order with high data reusability, to efficiently implement various advanced ME algorithms. The proposed ME-specific instructions can be selectively used for ME algorithms. The novel data-reusing search scan order, called center biased search scan (CBSS), exploits the symmetry of the search pattern to reduce redundant data loading on MESIP by about 26.9% and 16.1% compared with raster scan and snake scan, respectively. MESIP has been implemented with IBM's 90-nm CMOS technology and has 203 K gates excluding memory. Simulation results show that the proposed MESIP can reduce the number of required instructions by up to 18.9% compared with existing ME processors. Moreover, MESIP achieves comparable performance even with ME ASICs and hence may be quite suitable for a low-power and high-performance ME implementation. [ABSTRACT FROM AUTHOR]

Moving object detection is an important and fundamental step for intelligent video surveillance systems because it provides a focus of attention for post-processing. A multilayer codebook-based background subtraction (MCBS) model is proposed for video sequences to detect moving objects. Combining the multilayer block-based strategy and the adaptive feature extraction from blocks of various sizes, the proposed method can remove most of the nonstationary (dynamic) background and significantly increase the processing efficiency. Moreover, the pixel-based classification is adopted for refining the results from the block-based background subtraction, which can further classify pixels as foreground, shadows, and highlights. As a result, the proposed scheme can provide a high precision and efficient processing speed to meet the requirements of real-time moving object detection. [ABSTRACT FROM AUTHOR]

Binocular stereo matching is one of the most important algorithms in the field of computer vision. Adaptive support-weight approaches, the current state-of-the-art local methods, produce results comparable to those generated by global methods. However, excessive time consumption is the main problem of these algorithms since the computational complexity is proportionally related to the support window size. In this paper, we present a novel cost aggregation method inspired by domain transformation, a recently proposed dimensionality reduction technique. This transformation enables the aggregation of 2-D cost data to be performed using a sequence of 1-D filters, which lowers computation and memory costs compared to conventional 2-D filters. Experiments show that the proposed method outperforms the state-of-the-art local methods in terms of computational performance, since its computational complexity is independent of the input parameters. Furthermore, according to the experimental results with the Middlebury dataset and real-world images, our algorithm is currently one of the most accurate and efficient local algorithms. [ABSTRACT FROM AUTHOR]

Existing systems for 3-D reconstruction from multiple view video use controlled indoor environments with uniform illumination and backgrounds to allow accurate segmentation of dynamic foreground objects. In this paper, we present a portable system for 3-D reconstruction of dynamic outdoor scenes that require relatively large capture volumes with complex backgrounds and nonuniform illumination. This is motivated by the demand for 3-D reconstruction of natural outdoor scenes to support film and broadcast production. Limitations of existing multiple view 3-D reconstruction techniques for use in outdoor scenes are identified. Outdoor 3-D scene reconstruction is performed in three stages: 1) 3-D background scene modeling using spherical stereo image capture; 2) multiple view segmentation of dynamic foreground objects by simultaneous video matting across multiple views; and 3) robust 3-D foreground reconstruction and multiple view segmentation refinement in the presence of segmentation and calibration errors. Evaluation is performed on several outdoor productions with complex dynamic scenes including people and animals. Results demonstrate that the proposed approach overcomes limitations of previous indoor multiple view reconstruction approaches enabling high-quality free-viewpoint rendering and 3-D reference models for production. [ABSTRACT FROM AUTHOR]

This paper focuses on the problem of structure and motion recovery from uncalibrated image sequences. It has been empirically proven that image measurement uncertainties can be modeled spatially and temporally by virtue of reprojection residuals. Consequently, a spatial-and-temporal-weighted factorization (STWF) algorithm is proposed to handle significant noise contained in the tracking data. This paper presents three novelties and contributions. First, the image reprojection residual of a feature point is demonstrated to be generally proportional to the error magnitude associated with the image point. Second, the error distributions are estimated from a different perspective, that of the reprojection residuals. The image errors are modeled both spatially and temporally to cope with different kinds of uncertainties. Previous studies have considered only the spatial information. Third, based on the estimated error distributions, an STWF algorithm is proposed to improve the overall accuracy and robustness of traditional approaches. Unlike existing approaches, the proposed technique does not require prior information of image measurement and is easy to implement. Extensive experiments on synthetic data and real images validate the proposed method. [ABSTRACT FROM AUTHOR]

Recently, compressive sensing (CS) has emerged as a powerful tool for solving a class of inverse/underdetermined problems in computer vision and image processing. In this paper, we investigate the application of CS paradigms on single image super-resolution (SR) problems that are considered to be the most challenging in this class. In light of recent promising results, we propose novel tools for analyzing sparse representation-based inverse problems using redundant dictionary basis. Further, we provide novel results establishing tighter correspondence between SR and CS. As such, we gain insights into questions concerning regularizing the solution to the underdetermined problem, such as follows. 1) Is sparsity prior alone sufficient? 2) What is a good dictionary? 3) What is the practical implication of noncompliance with theoretical CS hypothesis? Unlike in other underdetermined problems that assume random down-projections, the low-resolution image formation model employed in CS-based SR is a deterministic down-projection that may not necessarily satisfy some critical assumptions of CS. We further investigate the impact of such projections in concern to the above questions. [ABSTRACT FROM PUBLISHER]

The research context of this paper is dynamic texture analysis and characterization. Many dynamic textures can be modeled as large scale propagating wavefronts and local oscillating phenomena. After introducing a formal model for dynamic textures, the morphological component analysis (MCA) approach with a well-chosen dictionary is used to retrieve the components of dynamic textures. We define two new strategies for adaptive thresholding in the MCA framework, which greatly reduce the computation time when applied on videos. Tests on real image sequences illustrate the efficiency of the proposed method. An application to global motion estimation is proposed and future prospects are finally exposed. [ABSTRACT FROM PUBLISHER]

Selective change-driven (SCD) vision is a capture/processing strategy especially suited for vision systems with limited resources and/or vision applications with real-time constraints. SCD vision capture essentially involves delivering only the pixels that have undergone the greatest change in illumination since the last time they were read-out. SCD vision processing involves processing a limited pixel flow with similar results to the usual image flow, but with far lower bandwidth and processing requirements. SCD vision is based on pixel flow processing instead of traditional image flow processing. This complete change in the way video is processed and has a direct impact on the processing hardware required to deal with visual information. In this paper, we present the first CMOS sensor using the SCD strategy, along with a highly resource-limited system implementing an object tracking experiment. Results show that SCD vision outperforms traditional vision systems by at least one order of magnitude, with limited hardware requirements for the specific tracking experiment being tested. [ABSTRACT FROM AUTHOR]

Faced with the growing population of seniors, developed countries need to establish new healthcare systems to ensure the safety of elderly people at home. Computer vision provides a promising solution to analyze personal behavior and detect certain unusual events such as falls. In this paper, a new method is proposed to detect falls by analyzing human shape deformation during a video sequence. A shape matching technique is used to track the person's silhouette along the video sequence. The shape deformation is then quantified from these silhouettes based on shape analysis methods. Finally, falls are detected from normal activities using a Gaussian mixture model. This paper has been conducted on a realistic data set of daily activities and simulated falls, and gives very good results (as low as 0% error with a multi-camera setup) compared with other common image processing methods. [ABSTRACT FROM AUTHOR]

Automated fire detection is an active research topic in computer vision. In this paper, we propose and analyze a new method for identifying fire in videos. Computer vision-based fire detection algorithms are usually applied in closed-circuit television surveillance scenarios with controlled background. In contrast, the proposed method can be applied not only to surveillance but also to automatic video classification for retrieval of fire catastrophes in databases of newscast content. In the latter case, there are large variations in fire and background characteristics depending on the video instance. The proposed method analyzes the frame-to-frame changes of specific low-level features describing potential fire regions. These features are color, area size, surface coarseness, boundary roughness, and skewness within estimated fire regions. Because of flickering and random characteristics of fire, these features are powerful discriminants. The behavioral change of each one of these features is evaluated, and the results are then combined according to the Bayes classifier for robust fire recognition. In addition, a priori knowledge of fire events captured in videos is used to significantly improve the classification results. For edited newscast videos, the fire region is usually located in the center of the frames. This fact is used to model the probability of occurrence of fire as a function of the position. Experiments illustrated the applicability of the method. [ABSTRACT FROM AUTHOR]

Optical flow is an important problem in computer vision since applications of accurate optical flow estimation enable us to control and manipulate tracking, 3-D reconstruction, motion blurring, and dirt removal. Many powerful methods have been proposed to solve the optical flow problem; however, instabilities at the boundaries of moving objects are still challenges. A difficult part of the optical flow problem is how to accurately and quickly detect and readjust unstable regions at the boundaries. This paper aims to enhance optical flow estimation by detecting and readjusting the unstable regions. In this paper, a new algorithm to detect and quickly readjust unstable regions at the boundaries of moving objects is presented in a more general and compact manner. In addition, a new context-based anisotropic diffusion filter, which is significant in processing intermediate data, is discussed in detail. Our approach has demonstrated more accurate results than previous approaches. [ABSTRACT FROM AUTHOR]