Search

Very deep Convolutional Neural Networks (CNNs) have greatly improved the performance on various image restoration tasks. However, this comes at a price of increasing computational burden, hence limiting their practical usages. We observe that some corrupted image regions are inherently easier to restore than others since the distortion and content vary within an image. To leverage this, we propose Path-Restore, a multi-path CNN with a pathfinder that can dynamically select an appropriate route for each image region. We train the pathfinder using reinforcement learning with a difficulty-regulated reward. This reward is related to the performance, complexity and “the difficulty of restoring a region”. A policy mask is further investigated to jointly process all the image regions. We conduct experiments on denoising and mixed restoration tasks. The results show that our method achieves comparable or superior performance to existing approaches with less computational cost. In particular, Path-Restore is effective for real-world denoising, where the noise distribution varies across different regions on a single image. Compared to the state-of-the-art RIDNet [1], our method achieves comparable performance and runs 2.7x faster on the realistic Darmstadt Noise Dataset [2]. Models and codes are available on the project page: https://www.mmlab-ntu.com/project/pathrestore/. [ABSTRACT FROM AUTHOR]

In this paper, we develop a new video-to-video face recognition algorithm. The major advantage of the video-based method is that more information is available in a video sequence than in a single image. In order to take advantage of the large amount of information in the video sequence and at the same time overcome the processing speed and data size problems, we develop several new techniques including temporal and spatial frame synchronization, multilevel discriminant subspace analysis, and multiclassifier integration for video sequence processing. An aligned video sequence for each person is first obtained by applying temporal and spatial synchronization, which effectively establishes the face correspondence using both audio and video information; then multilevel discriminant subspace analysis or multiclassifier integration is employed for further analysis based on the synchronized sequence. The method preserves most of the temporal-spatial information contained in a video sequence. Extensive experiments on the XM2VTS database clearly show the superiority of our new algorithms with near-perfect classification results (99.3%) obtained. Index Terms--Face recognition, subspace analysis, video processing.

In this paper, we develop a new framework for face recognition based on nonparametric discriminant analysis (NDA) and multiclassifier integration. Traditional LDA-based methods surfer a fundamental limitation originating from the parametric nature of scatter matrices, which are based on the Gaussian distribution assumption. The performance of these methods notably degrades when the actual distribution is non-Gaussian. To address this problem, we propose a new formulation of scatter matrices to extend the two-class NDA to multiclass cases. Then, in order to exploit the discriminant information in both the principal space and the null space of the intraclass scatter matrix, we develop two improved multiclass NDA-based algorithms (NSA and NFA) with each one having two complementary methods that are based on the principal space and the null space of the intraclass scatter matrix, respectively. Comparing to the NSA, the NFA is more effective in the utilization of the classification boundary information, in order to exploit the complementary nature of the two kinds of NFA (PNFA and NNFA), we finally develop a dual NFA-based multiclassifier fusion framework by employing the overcomplete Gabor representation for face images to boost the recognition performance. We show the improvements of the developed new algorithms over the traditional subspace methods through comparative experiments on two challenging face databases, the Purdue AR database and the XM2VTS database. Index Terms--Face recognition, classifier design and evaluation, nonparametric discriminant analysis (NDA), multiclassifier fusion.

In computer vision, shape matching is a challenging problem, especially when articulation and deformation of parts occur. These variations may be insignificant for human recognition but often cause a matching algorithm to give results that ate inconsistent with our perception. In this paper, we propose a novel shape descriptor of planar contours, called contour flexibility, which represents the deformable potential at each point along a contour. With this descriptor, the local and global features can be obtained from the contour. We then present a shape matching scheme based on the features obtained. Experiments with comparisons to recently published algorithms show that our algorithm performs best. Index Terms--2D shape, contour flexibility, matching.

The human vision system can interpret a single 2D line drawing as a 3D object without much difficulty even if the hidden lines of the object are invisible. Many reconstruction methods have been proposed to emulate this ability, but they cannot recover the complete object if the hidden lines of the object are not shown. This paper proposes a novel approach to reconstructing a complete 3D object, including the shape of the back of the object, from a line drawing without hidden lines. First, we develop theoretical constraints and an algorithm for the inference of the topology of the invisible edges and vertices of an object. Then, we present a reconstruction method based on perceptual symmetry and planarity of the object. We show a number of examples to demonstrate the success of our approach. Index Terms--3D reconstruction, hidden topology, line drawings, visual perception.

In previous optimization-based methods of 3D planar-faced object reconstruction from single 2D line drawings, the missing depths of the vertices of a line drawing (and other parameters in some methods) are used as the variables of the objective functions. A 3D object with planar faces is derived by finding values for these variables that minimize the objective functions. These methods work well for simple objects with a small number N of variables. As N grows, however, it is very difficult for them to find the expected objects. This is because with the nonlinear objective functions in a space of large dimension N, the search for optimal solutions can easily get trapped into local minima. In this paper, we use the parameters of the planes that pass through the planar faces of an object as the variables of the objective function. This leads to a set of linear constraints on the planes of the object, resulting in a much lower dimensional null space where optimization is easier to achieve. We prove that the dimension of this null space is exactly equal to the minimum number of vertex depths that define the 3D object. Since a practical line drawing is usually not an exact projection of a 3D object, we expand the null space to a larger space based on the singular value decomposition of the projection matrix of the line drawing. In this space, robust 3D reconstruction can be achieved. Compared with the two most related methods, our method not only can reconstruct more complex 3D objects from 2D line drawings but also is computationally more efficient. Index Terms--3D object reconstruction, degree of reconstruction freedom, line drawing, null space, singular value decomposition.

With many potential industrial applications, content-based image retrieval (CBIR) has recently gained more attention for image management and web searching. As an important tool to capture users' preferences and thus to improve the performance of CBIR systems, a variety of relevance feedback (RF) schemes have been developed in recent years. One key issue in RF is: which features (or feature dimensions) can benefit this human-computer iteration procedure? In this paper, we make theoretical and practical comparisons between principal and complement components of image features in CBIR RF. Most of the previous RF approaches treat the positive and negative feedbacks equivalently although this assumption is not appropriate since the two groups of training feedbacks have very different properties. That is, all positive feedbacks share a homogeneous concept while negative feedbacks do not. We explore solutions to this important problem by proposing an orthogonal complement component analysis. Experimental results are reported on a real-world image collection to demonstrate that the proposed complement components method consistently outperforms the conventional principal components method in both linear and kernel spaces when users want to retrieve images with a homogeneous concept. Index Terms--Content-based image retrieval (CBIR), kernel machine, orthogonal complement component analysis (OCCA), relevance feedback (RF), support vector machine (SVM).

This paper presents a unified criterion, Fisher + kernel criterion (FKC), for feature extraction and recognition. This new criterion is intended to extract the most discriminant features in different nonlinear spaces, and then, fuse these features under a unified measurement. Thus, FKC can simultaneously achieve nonlinear discriminant analysis and kernel selection. In addition, we present an efficient algorithm Fisher + kernel analysis (FKA), which utilizes the bilinear analysis, to optimize the new criterion. This FKA algorithm can alleviate the ill-posed problem existed in traditional kernel discriminant analysis (KDA), and usually, has no singularity problem. The effectiveness of our proposed algorithm is validated by a series of face-recognition experiments on several different databases. Index Terms--Bilinear analysis, discriminant analysis, face recognition, feature extraction, Fisher criterion, kernel selection.

A novel patch-based correspondence model is presented in this paper. Many segment-based correspondence approaches have been proposed in recent years. Untextured pixels and boundaries of discontinuities are imposed with hard constraints by the discontinuity assumption that large disparity variation only happens at the boundaries of segments in the above approaches. Significant improvements on performance of untextured and discontinuity area have been reported. But, the performance near occlusion is not satisfactory because a segmented region in one image may be only partially visible in the other one. To solve this problem, we utilize the observation that the shared edge of a visible area and an occluded area corresponds to the discontinuity in the other image. So, the proposed model conducts color segmentation on both images first and then a segment in one image is further cut into smaller patches corresponding to the boundaries of segments in the other when it is assigned with a disparity. Different visibility of patches in one segment is allowed. The uniqueness constraint in a segment level is used to compute the occlusions. An energy minimization framework using graph-cuts is proposed to find a global optimal configuration including both disparities and occlusions. Besides, some measurements are taken to make our segment-based algorithm suffer less from violation of the discontinuity assumption. Experimental results have shown superior performance of the proposed approach, especially on occlusions, untextured areas, and near discontinuities. Index Terms--Stereo, correspondence, segmentation, graph-cuts, occlusion, energy minimization.