Search

Small moving objects at far distance always occupy only one or a few pixels in image and exhibit extremely limited visual features, which bring great challenges to motion detection. Highly evolved visual systems endow flying insects with remarkable ability to pursue tiny mates and prey, providing a good template to develop image processing method for small target motion detection. The insects’ excellent sensitivity to small moving objects is believed to come from a class of specific neurons called small target motion detectors (STMDs). However, existing STMD-based methods often experience performance degradation when coping with complex natural scenes. In this paper, we propose a bio-inspired visual system with spatio-temporal feedback mechanism (called Spatio-Temporal Feedback STMD) to suppress false positive background movement while enhancing system responses to small targets. Specifically, the proposed visual system is composed of two complementary subnetworks and a feedback loop. The first subnetwork is designed to extract spatial and temporal movement patterns of cluttered background by neuronal ensemble coding. The second subnetwork is developed to capture small target motion information where its output and signal from the first subnetwork are integrated together via the feedback loop to filter out background false positives in a recurrent manner. Experimental results demonstrate that the proposed spatio-temporal feedback visual system is more competitive than existing methods in discriminating small moving targets from complex natural environments. [ABSTRACT FROM AUTHOR]

Face hallucination or super-resolution is a practical application of general image super-resolution which has been recently studied by many researchers. The challenge of good face hallucination comes from a variety of poses, illuminations, facial expressions, and other degradations. In many proposed methods, researchers resolve it by using a generative neural network to reduce the perceptual loss so we can generate a photo-realistic image. The problem is that researchers usually overlook the fidelity of the super-resolved image which could affect further facial image processing. Meanwhile, many CNN based approaches cascade multiple networks to extract facial prior information to improve super-resolution quality. Because of the end-to-end design, the details are missing for investigation. In this paper, we combine new techniques in convolutional neural network and random forests to a Hierarchical CNN based Random Forests (HCRF) approach for face super-resolution in a coarse-to-fine manner. In the proposed approach, we focus on a general approach that can handle facial images with various conditions without pre-processing. To the best of our knowledge, this is the first paper that combines the advantages of deep learning with random forests for face super-resolution. To achieve superior performance, we propose two novel CNN models for coarse facial image super-resolution and segmentation and then apply new random forests to target on local facial features refinement making use of the segmentation results. Extensive benchmark experiments on subjective and objective evaluation show that HCRF can achieve comparable speed and competitive performance compared with state-of-the-art super-resolution approaches for very low-resolution images. [ABSTRACT FROM AUTHOR]

Deep generative models have proven to be effective priors for solving a variety of image processing problems. However, the learning of realistic image priors, based on a large number of parameters, requires a large amount of training data. It has been shown recently, with the so-called deep image prior (DIP), that randomly initialized neural networks can act as good image priors without learning. In this paper, we propose a deep generative model for light fields, which is compact and which does not require any training data other than the light field itself. To show the potential of the proposed generative model, we develop a complete light field compression scheme with quantization-aware learning and entropy coding of the quantized weights. Experimental results show that the proposed method yields very competitive results compared with state-of-the-art light field compression methods, both in terms of PSNR and MS-SSIM metrics. [ABSTRACT FROM AUTHOR]

Arbitrary style transfer aims at migrating the style of a reference style painting to a target content image. Existing methods find it challenging to achieve good content fidelity and style migration at the same time. Moreover, they all rely on manually defined content and style, which is of limited universality and robustness. In this paper, we propose to introduce contrastive learning into style transfer, instructing the network to automatically learn to model the structural content and artistic style based on natural contrastive relationships in style transfer. Compared with existing methods, our learned modeling of content and style is more robust and universal. In addition, we further propose instance-wise contrastive style losses and a patch-wise contrastive content loss to guide style transfer. Combining the proposed contrastive losses and two self-reconstruction strategies, we develop a new style transfer framework, which is pluggable and can be flexibly applied to various style transfer modules. Experimental results demonstrate that our method has strong flexibility and synthesizes stylized images with higher quality. [ABSTRACT FROM AUTHOR]

Compressive covariance estimation has arisen as a class of techniques whose aim is to obtain second-order statistics of stochastic processes from compressive measurements. Recently, these methods have been used in various image processing and communications applications, including denoising, spectrum sensing, and compression. Notice that estimating the covariance matrix from compressive samples leads to ill-posed minimizations with severe performance loss at high compression rates. In this regard, a regularization term is typically aggregated to the cost function to consider prior information about a particular property of the covariance matrix. Hence, this paper proposes an algorithm based on the projected gradient method to recover low-rank or Toeplitz approximations of the covariance matrix from compressive measurements. The proposed algorithm divides the compressive measurements into data subsets projected onto different subspaces and accurately estimates the covariance matrix by solving a single optimization problem assuming that each data subset contains an approximation of the signal statistics. Furthermore, gradient filtering is included at every iteration of the proposed algorithm to minimize the estimation error. The error induced by the proposed splitting approach is analytically derived along with the convergence guarantees of the proposed method. The proposed algorithm estimates the covariance matrix of hyperspectral images from synthetic and real compressive samples. Extensive simulations show that the proposed algorithm can effectively recover the covariance matrix of hyperspectral images from compressive measurements with high compression ratios ($8-15\%$ approx) in noisy scenarios. Moreover, simulations and theoretical results show that the filtering step reduces the recovery error up to twice the number of eigenvectors. Finally, an optical implementation is proposed, and real measurements are used to validate the theoretical findings. [ABSTRACT FROM AUTHOR]

Serving as an essential step for many applications of image processing, superpixel generation has attracted a lot of attentions. Most existing superpixel generation algorithms focus on the boundary adherence and compactness of the superpixels, but ignore the topological consistency between the superpixels, which severely limites their applications in the subsequent tasks, especially in the CNN based image processing tasks. In this paper, we present a fast lattice superpixel generation algorithm, which can generate superpixels with lattice topology like the original pixels. We also propose a local similarity loss function to improve the segmentation accuracy of the generated lattice superpixels. The whole algorithm is parallelly implemented on GPU. We perform extensive experiments on three datasets (i.e., BSDS500, NYUv2 and VOC) to verify the efficacy of our algorithm. The experimental results show that our method achieves competitive results compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Given a photo of a subject, ability to generate a caricature image that captures distinct characteristics of the subject but with certain exaggeration of their prominent features is of fundamental importance to image processing and facial recognition. There are two main challenges in this task: shape exaggeration and style transfer. The former morphs and exaggerates key facial features of the subject, while the latter generates caricature images in a certain artistic style. In this paper, we propose a CAricature Style Transfer (CAST) framework for caricature generation. There are two modules in the proposed framework. The first is a geometric warping module. Different from the existing style transfer methods, we incorporate the Whitening and Coloring Transformation (WCT) in the geometric style transfer. The WCT is learned on photo and caricature landmarks or the caricature landmark space of a specific artist and is capable of transforming input photo landmarks to caricature landmarks. The second module is a texture style rendering module. We propose a new style transfer method by considering a semantic region-aligned style transfer via affinity constraint. Given a reference caricature image as the style reference, this module is capable of transferring styles between the same or similar semantic regions in caricatures and photos. Furthermore, it can transfer visual attributes of the reference caricatures (such as mouth shape and expressions) to the output caricatures. Experiments have shown desirable effects of the proposed method in transferring both the geometric and artistic texture styles of caricatures. Both qualitative and quantitative results show that the CAST framework is more effective compared than the state-of-the-art caricature generation methods. [ABSTRACT FROM AUTHOR]

Convolutional dictionary learning has become increasingly popular in signal and image processing for its ability to overcome the limitations of traditional patch-based dictionary learning. Although most studies on convolutional dictionary learning mainly focus on the unimodal case, real-world image processing tasks usually involve images from multiple modalities, e.g., visible and near-infrared (NIR) images. Thus, it is necessary to explore convolutional dictionary learning across different modalities. In this paper, we propose a novel multi-modal convolutional dictionary learning algorithm, which efficiently correlates different image modalities and fully considers neighborhood information at the image level. In this model, each modality is represented by two convolutional dictionaries, in which one dictionary is for common feature representation and the other is for unique feature representation. The model is constrained by the requirement that the convolutional sparse representations (CSRs) for the common features should be the same across different modalities, considering that these images are captured from the same scene. We propose a new training method based on the alternating direction method of multipliers (ADMM) to alternatively learn the common and unique dictionaries in the discrete Fourier transform (DFT) domain. We show that our model converges in less than 20 iterations between the convolutional dictionary updating and the CSRs calculation. The effectiveness of the proposed dictionary learning algorithm is demonstrated on various multimodal image processing tasks, achieves better performance than both dictionary learning methods and deep learning based methods with limited training data. [ABSTRACT FROM AUTHOR]

In this paper, we make the first benchmark effort to elaborate on the superiority of using RAW images in the low light enhancement and develop a novel alternative route to utilize RAW images in a more flexible and practical way. Inspired by a full consideration on the typical image processing pipeline, we are inspired to develop a new evaluation framework, Factorized Enhancement Model (FEM), which decomposes the properties of RAW images into measurable factors and provides a tool for exploring how properties of RAW images affect the enhancement performance empirically. The empirical benchmark results show that the Linearity of data and Exposure Time recorded in meta-data play the most critical role, which brings distinct performance gains in various measures over the approaches taking the sRGB images as input. With the insights obtained from the benchmark results in mind, a RAW-guiding Exposure Enhancement Network (REENet) is developed, which makes trade-offs between the advantages and inaccessibility of RAW images in real applications in a way of using RAW images only in the training phase. REENet projects sRGB images into linear RAW domains to apply constraints with corresponding RAW images to reduce the difficulty of modeling training. After that, in the testing phase, our REENet does not rely on RAW images. Experimental results demonstrate not only the superiority of REENet to state-of-the-art sRGB-based methods and but also the effectiveness of the RAW guidance and all components. [ABSTRACT FROM AUTHOR]

Recently, food recognition has received more and more attention in image processing and computer vision for its great potential applications in human health. Most of the existing methods directly extracted deep visual features via convolutional neural networks (CNNs) for food recognition. Such methods ignore the characteristics of food images and are, thus, hard to achieve optimal recognition performance. In contrast to general object recognition, food images typically do not exhibit distinctive spatial arrangement and common semantic patterns. In this paper, we propose a multi-scale multi-view feature aggregation (MSMVFA) scheme for food recognition. MSMVFA can aggregate high-level semantic features, mid-level attribute features, and deep visual features into a unified representation. These three types of features describe the food image from different granularity. Therefore, the aggregated features can capture the semantics of food images with the greatest probability. For that solution, we utilize additional ingredient knowledge to obtain mid-level attribute representation via ingredient-supervised CNNs. High-level semantic features and deep visual features are extracted from class-supervised CNNs. Considering food images do not exhibit distinctive spatial layout in many cases, MSMVFA fuses multi-scale CNN activations for each type of features to make aggregated features more discriminative and invariable to geometrical deformation. Finally, the aggregated features are more robust, comprehensive, and discriminative via two-level fusion, namely multi-scale fusion for each type of features and multi-view aggregation for different types of features. In addition, MSMVFA is general and different deep networks can be easily applied into this scheme. Extensive experiments and evaluations demonstrate that our method achieves state-of-the-art recognition performance on three popular large-scale food benchmark datasets in Top-1 recognition accuracy. Furthermore, we expect this paper will further the agenda of food recognition in the community of image processing and computer vision. [ABSTRACT FROM AUTHOR]

This paper aims at proposing a novel 2D non-linear phase decomposition of images, which performs the image processing tasks better than the traditional Fourier transformation (linear phase decomposition), but further, it has additional mathematical properties allowing more effective image analysis, including adaptive decomposition components and positive instantaneous phase derivatives. 1D unwinding Blaschke decomposition has recently been proposed and studied. Through factorization it expresses arbitrary 1D signal into an infinite linear combination of Blaschke products. It offers fast converging positive frequency decomposition in the form of rational approximation. However, in the multi-dimensional cases, the usual factorization mechanism does not work. As a consequence, there is no genuine unwinding decomposition for multi-dimensions. In this paper, a 2D partial unwinding decomposition based on algebraic transforms reducing multi-dimensions to the 1D case is proposed and analyzed. The result shows that the fast convergence offers efficient image reconstruction. The tensor type decomposing terms are mutually orthogonal, giving rise to 2D positive frequency decomposition. The comparison results show that the proposed method outperforms the standard greedy algorithm and the most commonly used methods in the Fourier category. An application in watermarking is presented to demonstrate its potential in applications. [ABSTRACT FROM AUTHOR]

Multi-view clustering aims at simultaneously obtaining a consensus underlying subspace across multiple views and conducting clustering on the learned consensus subspace, which has gained a variety of interest in image processing. In this paper, we propose the Semi-supervised Structured Subspace Learning algorithm for clustering data points from Multiple sources (SSSL-M). We explicitly extend the traditional multi-view clustering with a semi-supervised manner and then build an anti-block-diagonal indicator matrix with small amount of supervisory information to pursue the block-diagonal structure of the shared affinity matrix. SSSL-M regularizes multiple view-specific affinity matrices into a shared affinity matrix based on reconstruction through a unified framework consisting of backward encoding networks and the self-expressive mapping. The shared affinity matrix is comprehensive and can flexibly encode complementary information from multiple view-specific affinity matrices. An enhanced structural consistency of affinity matrices from different views can be achieved and the intrinsic relationships among affinity matrices from multiple views can be effectively reflected in this manner. Technically, we formulate the proposed model as an optimization problem, which can be solved by an alternating optimization scheme. Experimental results over seven different benchmark datasets demonstrate that better clustering results can be obtained by our method compared with the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

This work considers noise removal from images, focusing on the well-known K-SVD denoising algorithm. This sparsity-based method was proposed in 2006, and for a short while it was considered as state-of-the-art. However, over the years it has been surpassed by other methods, including the recent deep-learning-based newcomers. The question we address in this paper is whether K-SVD was brought to its peak in its original conception, or whether it can be made competitive again. The approach we take in answering this question is to redesign the algorithm to operate in a supervised manner. More specifically, we propose an end-to-end deep architecture with the exact K-SVD computational path, and train it for optimized denoising. Our work shows how to overcome difficulties arising in turning the K-SVD scheme into a differentiable, and thus learnable, machine. With a small number of parameters to learn and while preserving the original K-SVD essence, the proposed architecture is shown to outperform the classical K-SVD algorithm substantially, and getting closer to recent state-of-the-art learning-based denoising methods. Adopting a broader context, this work touches on themes around the design of deep-learning solutions for image processing tasks, while paving a bridge between classic methods and novel deep-learning-based ones. [ABSTRACT FROM AUTHOR]

Image processing in hexagonal lattice has many advantages rather than square lattice. Researchers have addressed benefits of hexagonal structure in applications such as binarization, rotation, scaling and edge detection. Approximately all existing hardwares for capturing and displaying images are based on square lattice. Therefore, the best way for using advantages of hexagonal lattice is to find a proper software approach to convert square pixels to hexagonal ones. This paper presents a hexagonal platform based on interpolation which addresses three existing hexagonal challenges including imperfect hexagonal shape, inaccurate intensity level of hexagonal pixels and lower resolution in hexagonal space. The proposed interpolation is computed by overlaps between square and hexagonal pixels. Overlap types are formulated mathematically in 8 separate cases. Each overlap case is detected automatically and used to compute final gray-level intensity of hexagonal pixels. It is mathematically and experimentally shown that the proposed method satisfies necessary conditions for square-to-hexagonal conversion. The proposed scheme is evaluated on synthetic and real images with 10 different levels of noise in interpolation and edge detection applications. In synthetic images, the proposed method achieves the best figure of merit (FOM) 99.92% and 98.67% in high and low SNRs 100 and 20, respectively. Also, the proposed method outperforms existing state of the art hexagonal lattices with interclass correlation coefficient (ICC) 84.18% and mean rating 7.7 (out of 9) in real images. [ABSTRACT FROM AUTHOR]

Inverse problems are a group of important mathematical problems that aim at estimating source data $x$ and operation parameters $z$ from inadequate observations $y$. In the image processing field, most recent deep learning-based methods simply deal with such problems under a pixel-wise regression framework (from $y$ to $x$) while ignoring the physics behind. In this paper, we re-examine these problems under a different viewpoint and propose a novel framework for solving certain types of inverse problems in image processing. Instead of predicting $x$ directly from $y$ , we train a deep neural network to estimate the degradation parameters $z$ under an adversarial training paradigm. We show that if the degradation behind satisfies some certain assumptions, the solution to the problem can be improved by introducing additional adversarial constraints to the parameter space and the training may not even require pair-wise supervision. In our experiment, we apply our method to a variety of real-world problems, including image denoising, image deraining, image shadow removal, non-uniform illumination correction, and underdetermined blind source separation of images or speech signals. The results on multiple tasks demonstrate the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Traditional image signal processing (ISP) pipeline consists of a set of cascaded image processing modules onboard a camera to reconstruct a high-quality sRGB image from the sensor raw data. Recently, some methods have been proposed to learn a convolutional neural network (CNN) to improve the performance of traditional ISP. However, in these works usually a CNN is directly trained to accomplish the ISP tasks without considering much the correlation among the different components in an ISP. As a result, the quality of reconstructed images is barely satisfactory in challenging scenarios such as low-light imaging. In this paper, we firstly analyze the correlation among the different tasks in an ISP, and categorize them into two weakly correlated groups: restoration and enhancement. Then we design a two-stage network, called CameraNet, to progressively learn the two groups of ISP tasks. In each stage, a ground truth is specified to supervise the subnetwork learning, and the two subnetworks are jointly fine-tuned to produce the final output. Experiments on three benchmark datasets show that the proposed CameraNet achieves consistently compelling reconstruction quality and outperforms the recently proposed ISP learning methods. [ABSTRACT FROM AUTHOR]

Plant disease diagnosis is very critical for agriculture due to its importance for increasing crop production. Recent advances in image processing offer us a new way to solve this issue via visual plant disease analysis. However, there are few works in this area, not to mention systematic researches. In this paper, we systematically investigate the problem of visual plant disease recognition for plant disease diagnosis. Compared with other types of images, plant disease images generally exhibit randomly distributed lesions, diverse symptoms and complex backgrounds, and thus are hard to capture discriminative information. To facilitate the plant disease recognition research, we construct a new large-scale plant disease dataset with 271 plant disease categories and 220,592 images. Based on this dataset, we tackle plant disease recognition via reweighting both visual regions and loss to emphasize diseased parts. We first compute the weights of all the divided patches from each image based on the cluster distribution of these patches to indicate the discriminative level of each patch. Then we allocate the weight to each loss for each patch-label pair during weakly-supervised training to enable discriminative disease part learning. We finally extract patch features from the network trained with loss reweighting, and utilize the LSTM network to encode the weighed patch feature sequence into a comprehensive feature representation. Extensive evaluations on this dataset and another public dataset demonstrate the advantage of the proposed method. We expect this research will further the agenda of plant disease recognition in the community of image processing. [ABSTRACT FROM AUTHOR]

Superpixel segmentation, as a central image processing task, has many applications in computer vision and computer graphics. Boundary alignment and shape compactness are leading indicators to evaluate a superpixel segmentation algorithm. Furthermore, convexity can make superpixels reflect more geometric structures in images and provide a more concise over-segmentation result. In this paper, we consider generating convex and compact superpixels while satisfying the constraints of adhering to the boundary as far as possible. We formulate the new superpixel segmentation into an edge-constrained centroidal power diagram (ECCPD) optimization problem. In the implementation, we optimize the superpixel configurations by repeatedly performing two alternative operations, which include site location updating and weight updating through a weight function defined by image features. Compared with existing superpixel methods, our method can partition an image into fully convex and compact superpixels with better boundary adherence. Extensive experimental results show that our approach outperforms existing superpixel segmentation methods in boundary alignment and compactness for generating convex superpixels. [ABSTRACT FROM AUTHOR]

We present an approach to extend the functionality and the use of kernel machines in image processing applications. We introduce a novel way to design spatial kernel machines with spatial properties and demonstrate how those newly introduced spatial properties enhance the possibilities of the use of kernel machines in image processing applications as a proof of concept. In this paper, we demonstrate four particular extensions: 1) how to model shapes efficiently with spatially computed kernel parameters in a geometrically scalable way; 2) how to visualize the kernel parameters precisely and intuitively on binary 2D shapes; 3) how to construct a one-class classifier from the binary classifier in a straightforward manner without re-training; and 4) how to use the computed kernel parameters for filtering. The existing literature on kernel machines mostly focuses on estimating the optimal kernel parameters via additional cost function(s). In this paper, instead of employing an additional cost function to estimate the kernel-related parameters, we investigate on an analytical solution to predict the actual kernel parameters locally and show how to build a spatial kernel machine with our analytical approach. Classical kernel machines do not perform well on precise shape modeling with a low number of support vectors as demonstrated in this paper. However, we demonstrate and visualize that our analytical approach provides a natural means to relate the kernel parameters to the 2D shapes for sparse shape modeling, where the shape boundary represents the decision boundary. For that, we incorporate the selected kernel function’s geometric properties as an additional constraint into the classifier’s optimization problem by defining an easy-to-explain and intuitive concept: similarity domains. In our experiments, we study and demonstrate how the resulting new kernel machine enhances the capabilities of the classical kernel machines with applications on shape modeling, (geometrically) scaling the non-linear decision boundary at various scales and precise visualization of the kernel parameters in 2D images. [ABSTRACT FROM AUTHOR]

Identifying the underlying models in a set of data points that is contaminated by noise and outliers leads to a highly complex multi-model fitting problem. This problem can be posed as a clustering problem by the projection of higher-order affinities between data points into a graph, which can be clustered using spectral clustering. Calculating all possible higher-order affinities is computationally expensive. Hence, in most cases, only a subset is used. In this paper, we propose an effective sampling method for obtaining a highly accurate approximation of the full graph, which is required to solve multi-structural model fitting problems in computer vision. The proposed method is based on the observation that the usefulness of a graph for segmentation improves as the distribution of the hypotheses that are used to build the graph approaches the distribution of the actual parameters for the given data. In this paper, we approximate this actual parameter distribution by using a $k$ th-order statistics-based cost function, and the samples are generated using a greedy algorithm that is coupled with a data sub-sampling strategy. The experimental analysis shows that the proposed method is both accurate and computationally efficient compared with the state-of-the-art robust multi-model fitting techniques. The implementation of the method is publicly available from https://github.com/RuwanT/model-fitting-cbs. [ABSTRACT FROM AUTHOR]

Fog removal from an image is an active research topic in computer vision. However, current literature is weak in the following two areas which in many ways are hindering progress for developing defogging algorithms. First, there is no true real-world and naturally occurring foggy image datasets suitable for developing defogging models. Second, there is no suitable mathematically simple and easy to use image quality assessment (IQA) methods for evaluating the visual quality of defogged images. We address these two aspects in this paper. We first introduce a new foggy image dataset called multiple real-world foggy image dataset (MRFID). MRFID contains foggy and clear images of 200 outdoor scenes. For each scene, one clear image and 4 foggy images of different densities defined as slightly foggy, moderately foggy, highly foggy, and extremely foggy, are manually selected from images taken from these scenes over the course of one calendar year. We then process the foggy images of MRFID using 16 defogging methods to obtain 12,800 defogged images (DFIs) and perform a comprehensive subjective evaluation of the visual quality of the DFIs. Through collecting the mean opinion score (MOS) of 120 subjects and evaluating a variety of fog-relevant image features, we have developed a new Fog-relevant Feature based SIMilarity index (FRFSIM) for assessing the visual quality of DFIs. We present extensive experimental results to show that our new visual quality assessment measure, the FRFSIM, is more consistent with the MOS than other IQA methods and is therefore more suitable for evaluating defogged images than other state-of-the-art IQA methods. Our dataset and relevant code are available at http://www.vistalab.ac.cn/MRFID-for-defogging/. [ABSTRACT FROM AUTHOR]

Most of the previous studies on the user preferences assume that there is a personal transitive preference ranking of the consumable media like images. For example, the transitivity of preferences is one of the most important assumptions in the recommender system research. However, the intransitive relations have also been widely observed, such as the win/loss relations in online video games, in sport matches, and even in rock–paper–scissors games. It is also found that different subjects demonstrate the personalized intransitive preferences in the pairwise comparisons between the applicants for college admission. Since the intransitivity of preferences on images has barely been studied before and has a large impact on the research of personalized image search and recommendation, it is necessary to propose a novel method to predict the personalized intransitive preferences of images. In this paper, we propose the novel Multi-Criterion preference (MuCri) models to predict the intransitive relations in the image preferences. The MuCri models utilize different kinds of image content features as well as the latent features of users and images. Meanwhile, a new data set is constructed in this paper, in order to evaluate the performance of the MuCri models. The experimental evaluation shows that the MuCri models outperform all the baselines. Due to the interdisciplinary nature of this topic, we believe it would widely attract the attention of researchers in the image processing community as well as in other communities, such as machine learning, multimedia, and recommender system. [ABSTRACT FROM AUTHOR]

In this paper we combine video compression and modern image processing methods. Iterative filter methods for prediction signals based on classic inpainting methods are introduced and extensive parameter tests are described. In order to construct an alternative prediction filter for video coding, techniques originally employed for inpainting are applied. Thereby, the structures of the underlying prediction were incorporated into the filter construction making it signal adaptive. The resulting optimization problem is solved using the so-called Alternating Direction Method of Multipliers (ADMM). The undertaken novel parameter tests are described and it is shown that they improve the coding efficiency of the tool. The suggested filter is embedded into a software based on HEVC with additional QTBT (Quadtree plus Binary Tree) and MTT (Multi-Type-Tree) block structure. Overall, the proposed filter method obtains average bitrate savings of 1.35% at an average encoder runtime increase of 28% and decoder runtime increase of 38%. UHD test sequences achieve bitrate savings of up to 3.66% for Random Access. [ABSTRACT FROM AUTHOR]

Regularization-based image restoration is one of the most powerful tools in image processing and computer vision thanks to its flexibility for handling various inverse problems. However, designing an optimal regularization function still remains unsolved since natural images and related scene types have a complex structure. In this paper, we present a general and principled framework, called deeply aggregated alternating minimization (DeepAM). We design a convolutional neural network (CNN) to implicitly parameterize the regularizer of the alternating minimization (AM) algorithm. Contrary to the conventional AM algorithm based on a point-wise proximal mapping, the DeepAM projects intermediate estimate into a set of natural images via deep aggregation. Since the CNN is fully integrated into the AM procedure, all parameters can be jointly optimized through end-to-end training. These properties enable the DeepAM to converge with a small number of iterations, while maintaining an algorithmic simplicity. We show that the DeepAM outperforms state-of-the-art methods, including nonlocal-based methods, Plug-and-Play regularization, and recent data-driven approaches. The effectiveness of our framework is demonstrated in a variety of image restoration tasks: Guassian denoising, deraining, deblurring, super-resolution, color-guided depth upsampling, and RGB/NIR restoration. [ABSTRACT FROM AUTHOR]

Traditional image processing operators often provide some control parameters to tweak the final results. Recently, different convolutional neural networks have been used to approximate or improve these operators. However, in those methods, one single model can only handle one operator of a specific parameter value and does not support parameter tuning. In this paper, we propose a new plugin module, “Adaptive Filterbank Pyramid”, which can be inserted into a backbone network to support multiple operators and continuous parameter tuning. Our module explicitly represents one operator with one filterbank pyramid. To generate the results of a specific operator, the corresponding filterbank pyramid is convolved with the intermediate feature pyramid produced by the backbone network. The weights of the filterbank pyramid are directly regressed by another sub-network, which is jointly trained with the backbone network and adapted to the input parameter, thus enabling continuous parameter tuning. We applied the proposed module for a large variety of image processing tasks, including image smoothing, image denoising, image deblocking, image enhancement and neural style transfer. Experiments show that our method is generalized to different types of image processing tasks and different backbone network structures. Compared to the single-operator-single-parameter baseline, our method can produce comparable results but is significantly more efficient in both training and testing. [ABSTRACT FROM AUTHOR]

Low-light image enhancement is a challenging task that has attracted considerable attention. Pictures taken in low-light conditions often have bad visual quality. To address the problem, we regard the low-light enhancement as a residual learning problem that is to estimate the residual between low- and normal-light images. In this paper, we propose a novel Deep Lightening Network (DLN) that benefits from the recent development of Convolutional Neural Networks (CNNs). The proposed DLN consists of several Lightening Back-Projection (LBP) blocks. The LBPs perform lightening and darkening processes iteratively to learn the residual for normal-light estimations. To effectively utilize the local and global features, we also propose a Feature Aggregation (FA) block that adaptively fuses the results of different LBPs. We evaluate the proposed method on different datasets. Numerical results show that our proposed DLN approach outperforms other methods under both objective and subjective metrics. [ABSTRACT FROM AUTHOR]

Sparse representation has achieved great success in various image processing and computer vision tasks. For image processing, typical patch-based sparse representation (PSR) models usually tend to generate undesirable visual artifacts, while group-based sparse representation (GSR) models lean to produce over-smooth effects. In this paper, we propose a new sparse representation model, termed joint patch-group based sparse representation (JPG-SR). Compared with existing sparse representation models, the proposed JPG-SR provides an effective mechanism to integrate the local sparsity and nonlocal self-similarity of images. We then apply the proposed JPG-SR to image restoration tasks, including image inpainting and image deblocking. An iterative algorithm based on the alternating direction method of multipliers (ADMM) framework is developed to solve the proposed JPG-SR based image restoration problems. Experimental results demonstrate that the proposed JPG-SR is effective and outperforms many state-of-the-art methods in both objective and perceptual quality. [ABSTRACT FROM AUTHOR]

Generating text to have a consistent style with only a few observed highly-stylized text samples is a difficult task for image processing. The text style involving the typography, i.e., font, stroke, color, decoration, effects, etc., should be considered for transfer. In this paper, we propose a novel approach to stylize target text by decoding weighted deep features from only a few referenced samples. The deep features, including content and style features of each referenced text, are extracted from a Convolutional Neural Network (CNN) that is optimized for character recognition. Then, we calculate the similarity scores of the target text and the referenced samples by measuring the distance along the corresponding channels from the content features of the CNN when considering only the content, and assign them as the weights for aggregating the deep features. To enforce the stylized text to be realistic, a discriminative network with adversarial loss is employed. We demonstrate the effectiveness of our network by conducting experiments on three different datasets which have various styles, fonts, languages, etc. Additionally, the coefficients for character style transfer, including the character content, the effect of similarity matrix, the number of referenced characters, the similarity between characters, and performance evaluation by a new protocol are analyzed for better understanding our proposed framework. [ABSTRACT FROM AUTHOR]

Polarimetric synthetic aperture radar (PolSAR) image segmentation is currently of great importance in image processing for remote sensing applications. However, it is a challenging task due to two main reasons. Firstly, the label information is difficult to acquire due to high annotation costs. Secondly, the speckle effect embedded in the PolSAR imaging process remarkably degrades the segmentation performance. To address these two issues, we present a contextual PolSAR image semantic segmentation method in this paper. With a newly defined channel-wise consistent feature set as input, the three-dimensional discrete wavelet transform (3D-DWT) technique is employed to extract discriminative multi-scale features that are robust to speckle noise. Then Markov random field (MRF) is further applied to enforce label smoothness spatially during segmentation. By simultaneously utilizing 3D-DWT features and MRF priors for the first time, contextual information is fully integrated during the segmentation to ensure accurate and smooth segmentation. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on three real benchmark PolSAR image data sets. Experimental results indicate that the proposed method achieves promising segmentation accuracy and preferable spatial consistency using a minimal number of labeled pixels. [ABSTRACT FROM AUTHOR]

Ill-posed linear inverse problems appear in many image processing applications, such as deblurring, super-resolution and compressed sensing. Many restoration strategies involve minimizing a cost function, which is composed of fidelity and prior terms, balanced by a regularization parameter. While a vast amount of research has been focused on different prior models, the fidelity term is almost always chosen to be the least squares (LS) objective, that encourages fitting the linearly transformed optimization variable to the observations. In this paper, we examine a different fidelity term, which has been implicitly used by the recently proposed iterative denoising and backward projections (IDBP) framework. This term encourages agreement between the projection of the optimization variable onto the row space of the linear operator and the pseudo-inverse of the linear operator (“back-projection”) applied on the observations. We analytically examine the difference between the two fidelity terms for Tikhonov regularization and identify cases (such as a badly conditioned linear operator) where the new term has an advantage over the standard LS one. Moreover, we demonstrate empirically that the behavior of the two induced cost functions for sophisticated convex and non-convex priors, such as total-variation, BM3D, and deep generative models, correlates with the obtained theoretical analysis. [ABSTRACT FROM AUTHOR]

Image deraining is an important yet challenging image processing task. Though deterministic image deraining methods are developed with encouraging performance, they are infeasible to learn flexible representations for probabilistic inference and diverse predictions. Besides, rain intensity varies both in spatial locations and across color channels, making this task more difficult. In this paper, we propose a Conditional Variational Image Deraining (CVID) network for better deraining performance, leveraging the exclusive generative ability of Conditional Variational Auto-Encoder (CVAE) on providing diverse predictions for the rainy image. To perform spatially adaptive deraining, we propose a spatial density estimation (SDE) module to estimate a rain density map for each image. Since rain density varies across different color channels, we also propose a channel-wise (CW) deraining scheme. Experiments on synthesized and real-world datasets show that the proposed CVID network achieves much better performance than previous deterministic methods on image deraining. Extensive ablation studies validate the effectiveness of the proposed SDE module and CW scheme in our CVID network. The code is available at https://github.com/Yingjun-Du/VID. [ABSTRACT FROM AUTHOR]

Computing the convolution between a 2D signal and a corresponding filter with variable orientations is a basic problem that arises in various tasks ranging from low level image processing (e.g. ridge/edge detection) to high level computer vision (e.g. pattern recognition). Through decades of research, there still lacks an efficient method for solving this problem. In this paper, we investigate this problem from the perspective of approximation by considering the following problem: what is the optimal basis for approximating all rotated versions of a given bivariate function? Surprisingly, solely minimising the $L^{2}$ -approximation-error leads to a rotation-covariant linear expansion, which we name Fourier-Argand representation. This representation presents two major advantages: 1) rotation-covariance of the basis, which implies a “strong steerability” — rotating by an angle $\alpha $ corresponds to multiplying each basis function by a complex scalar $e^{-ik\alpha }$ ; 2) optimality of the Fourier-Argand basis, which ensures a few number of basis functions suffice to accurately approximate complicated patterns and highly direction-selective filters. We show the relation between the Fourier-Argand representation and the Radon transform, leading to an efficient implementation of the decomposition for digital filters. We also show how to retrieve accurate orientation of local structures/patterns using a fast frequency estimation algorithm. [ABSTRACT FROM AUTHOR]

This paper presents a new approach to recognizing vanishing-point-constrained building planes from a single image of street view. We first design a novel convolutional neural network (CNN) architecture that generates geometric segmentation of per-pixel orientations from a single street-view image. The network combines two-stream features of general visual cues and surface normals in gated convolution layers, and employs a deeply supervised loss that encapsulates multi-scale convolutional features. Our experiments on a new benchmark with fine-grained plane segmentations of real-world street views show that our network outperforms state-of-the-arts methods of both semantic and geometric segmentation. The pixel-wise segmentation exhibits coarse boundaries and discontinuities. We then propose to rectify the pixel-wise segmentation into perspectively-projected quads based on spatial proximity between the segmentation masks and exterior line segments detected through an image processing. We demonstrate how the results can be utilized to perspectively overlay images and icons on building planes in input photos, and provide visual cues for various applications. [ABSTRACT FROM AUTHOR]

Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We build a novel image fusion framework based on MDLatLRR which is used to decompose source images into detail parts(salient features) and base parts. A nuclear-norm based fusion strategy is used to fuse the detail parts and the base parts are fused by an averaging strategy. Compared with other state-of-the-art fusion methods, the proposed algorithm exhibits better fusion performance in both subjective and objective evaluation. [ABSTRACT FROM AUTHOR]

Lossy image compression (LIC), which aims to utilize inexact approximations to represent an image more compactly, is a classical problem in image processing. Recently, deep convolutional neural networks (CNNs) have achieved interesting results in LIC by learning an encoder-quantizer-decoder network from a large amount of data. However, existing CNN-based LIC methods generally train a network for a specific bits-per-pixel (bpp). Such a “one-network-per-bpp” problem limits the generality and flexibility of CNNs to practical LIC applications. In this paper, we propose to learn a single CNN which can perform LIC at multiple bpp rates. A simple yet effective Tucker Decomposition Network (TDNet) is developed, where there is a novel tucker decomposition layer (TDL) to decompose a latent image representation into a set of projection matrices and a core tensor. By changing the rank of core tensor and its quantization, we can easily adjust the bpp rate of latent image representation within a single CNN. Furthermore, an iterative non-uniform quantization scheme is presented to optimize the quantizer, and a coarse-to-fine training strategy is introduced to reconstruct the decompressed images. Extensive experiments demonstrate the state-of-the-art compression performance of TDNet in terms of both PSNR and MS-SSIM indices. [ABSTRACT FROM AUTHOR]

Structure-texture image decomposition is a fundamental but challenging topic in computational graphics and image processing. In this paper, we introduce a structure-aware and a texture-aware measures to facilitate the structure-texture decomposition (STD) of images. Edge strengths and spatial scales that have been widely-used in previous STD researches cannot describe the structures and textures of images well. The proposed two measures differentiate image textures from image structures based on their distinctive characteristics. Specifically, the first one aims to measure the anisotropy of local gradients, and the second one is designed to measure the repeatability degree of signal patterns in a neighboring region. Since these two measures describe different properties of image structures and textures, they are complementary to each other. The STD is achieved by optimizing an objective function based on the two new measures. As using traditional optimization methods to solve the optimization problem will require designing different optimizers for different functional spaces, we employ an architecture of deep neural network to optimize the STD cost function in a unified manner. The experimental results demonstrate that, as compared with some state-of-the-art methods, our method can better separate image structure and texture and result in shaper edges in the structural component. Furthermore, to demonstrate the usefulness of the proposed STD method, we have successfully applied it to several applications including detail enhancement, edge detection, and visual quality assessment of super-resolved images. [ABSTRACT FROM AUTHOR]

Eye tracking technology in low resolution scenarios is not a completely solved issue to date. The possibility of using eye tracking in a mobile gadget is a challenging objective that would permit to spread this technology to non-explored fields. In this paper, a knowledge based approach is presented to solve gaze estimation in low resolution settings. The understanding of the high resolution paradigm permits to propose alternative models to solve gaze estimation. In this manner, three models are presented: a geometrical model, an interpolation model and a compound model, as solutions for gaze estimation for remote low resolution systems. Since this work considers head position essential to improve gaze accuracy, a method for head pose estimation is also proposed. The methods are validated in an optimal framework, I2Head database, which combines head and gaze data. The experimental validation of the models demonstrates their sensitivity to image processing inaccuracies, critical in the case of the geometrical model. Static and extreme movement scenarios are analyzed showing the higher robustness of compound and geometrical models in the presence of user’s displacement. Accuracy values of about 3° have been obtained, increasing to values close to 5° in extreme displacement settings, results fully comparable with the state-of-the-art. [ABSTRACT FROM AUTHOR]

Connected Components Labeling is an essential step of many Image Processing and Computer Vision tasks. Since the first proposal of a labeling algorithm, which dates back to the sixties, many approaches have optimized the computational load needed to label an image. In particular, the use of decision forests and state prediction have recently appeared as valuable strategies to improve performance. However, due to the overhead of the manual construction of prediction states and the size of the resulting machine code, the application of these strategies has been restricted to small masks, thus ignoring the benefit of using a block-based approach. In this paper, we combine a block-based mask with state prediction and code compression: the resulting algorithm is modeled as a Directed Rooted Acyclic Graph with multiple entry points, which is automatically generated without manual intervention. When tested on synthetic and real datasets, in comparison with optimized implementations of state-of-the-art algorithms, the proposed approach shows superior performance, surpassing the results obtained by all compared approaches in all settings. [ABSTRACT FROM AUTHOR]

The paper presents the results from a multi-year effort to develop and validate image processing methods for selecting the best physical models based on solar image observations. The approach consists of selecting the physical models based on their agreement with coronal holes extracted from the images. Ultimately, the goal is to use physical models to predict geomagnetic storms. We decompose the problem into three subproblems: (i) coronal hole segmentation based on physical constraints, (ii) matching clusters of coronal holes between different maps, and (iii) physical map classification. For segmenting coronal holes, we develop a multi-modal method that uses segmentation maps from three different methods to initialize a level-set method that evolves the initial coronal hole segmentation to the magnetic boundary. Then, we introduce a new method based on Linear Programming for matching clusters of coronal holes. The final matching is then performed using Random Forests. The methods were carefully validated using consensus maps derived from multiple readers, manual clustering, manual map classification, and method validation for 50 maps. The proposed multi-modal segmentation method significantly outperformed SegNet, U-net, Henney-Harvey, and FCN by providing accurate boundary detection. Overall, the method gave a 95.5% map classification accuracy. [ABSTRACT FROM AUTHOR]

The guided filter and its subsequent derivatives have been widely employed in many image processing and computer vision applications primarily brought about by their low complexity and good edge-preservation properties. Despite this success, the different variants of the guided filter are unable to handle more aggressive filtering strengths leading to the manifestation of “detail halos”. At the same time, these existing filters perform poorly when the input and guide images have structural inconsistencies. In this paper, we demonstrate that these limitations are due to the guided filter operating as a variable-strength locally-isotropic filter that, in effect, acts as a weak anisotropic filter on the image. Our analysis shows that this behaviour stems from the use of unweighted averaging in the final steps of guided filter variants including the adaptive guided filter (AGF), weighted guided image filter (WGIF), and gradient-domain guided image filter (GGIF). We propose a novel filter, the Anisotropic Guided Filter (AnisGF), that utilises weighted averaging to achieve maximum diffusion while preserving strong edges in the image. The proposed weights are optimised based on the local neighbourhood variances to achieve strong anisotropic filtering while preserving the low computational cost of the original guided filter. Synthetic tests show that the proposed method addresses the presence of detail halos and the handling of inconsistent structures found in previous variants of the guided filter. Furthermore, experiments in scale-aware filtering, detail enhancement, texture removal, and chroma upsampling demonstrate the improvements brought about by the technique. [ABSTRACT FROM AUTHOR]

Image restoration (IR) is a long-standing challenging problem in low-level image processing. It is of utmost importance to learn good image priors for pursuing visually pleasing results. In this paper, we develop a multi-channel and multi-model-based denoising autoencoder network as image prior for solving IR problem. Specifically, the network that trained on RGB-channel images is used to construct a prior at first, and then the learned prior is incorporated into single-channel grayscale IR tasks. To achieve the goal, we employ the auxiliary variable technique to integrate the higher-dimensional network-driven prior information into the iterative restoration procedure. In addition, according to the weighted aggregation idea, a multi-model strategy is put forward to enhance the network stability that favors to avoid getting trapped in local optima. Extensive experiments on image deblurring and deblocking tasks show that the proposed algorithm is efficient, robust, and yields state-of-the-art restoration quality on grayscale images. [ABSTRACT FROM AUTHOR]

Delineation of curvilinear structures in images is an important basic step of several image processing applications, such as segmentation of roads or rivers in aerial images, vessels or staining membranes in medical images, and cracks in pavements and roads, among others. Existing methods suffer from insufficient robustness to noise. In this paper, we propose a novel operator for the detection of curvilinear structures in images, which we demonstrate to be robust to various types of noise and effective in several applications. We call it RUSTICO, which stands for RobUST Inhibition-augmented Curvilinear Operator. It is inspired by the push–pull inhibition in visual cortex and takes as input the responses of two trainable B-COSFIRE filters of opposite polarity. The output of RUSTICO consists of a magnitude map and an orientation map. We carried out experiments on a data set of synthetic stimuli with noise drawn from different distributions, as well as on several benchmark data sets of retinal fundus images, crack pavements, and aerial images and a new data set of rose bushes used for automatic gardening. We evaluated the performance of RUSTICO by a metric that considers the structural properties of line networks (connectivity, area, and length) and demonstrated that RUSTICO outperforms many existing methods with high statistical significance. RUSTICO exhibits high robustness to noise and texture. [ABSTRACT FROM AUTHOR]

A quick-response code (QR code) is a two-dimensional code akin to a barcode that encodes a message of limited length. In this paper, we present a variant of QR code, a two-layer QR code. Its two-layer structure can display two alternative messages when scanned from two different directions. We propose a method to generate such two-layer QR codes encoding two given messages in a few seconds. We also demonstrate the robustness of our method on both synthetic and fabricated examples. All source code will be made publicly available (https://github.com/yuantailing/two-layer-qrcode). [ABSTRACT FROM AUTHOR]

Curvilinear structure restoration in image processing procedures is a difficult task, which can be compounded when these structures are thin, i.e., when their smallest dimension is close to the resolution of the sensor. Many recent restoration methods involve considering a local gradient-based regularization term as prior, assuming gradient sparsity. An isotropic gradient operator is typically not suitable for thin curvilinear structures, since gradients are not sparse for these. In this paper, we propose a mixed gradient operator that combines a standard gradient in the isotropic image regions, and a directional gradient in the regions where specific orientations are likely. In particular, such information can be provided by curvilinear structure detectors (e.g., RORPO or Frangi filters). Our proposed mixed gradient operator, that can be viewed as a companion tool of such detectors, is proposed in a discrete framework and its formulation/computation holds in any dimension; in other words, it is valid in $\mathbb Z^{n}$ , ${n}\geq 1$. We show how this mixed gradient can be used to construct image priors that take edge orientation, as well as intensity, into account, and then involved in various image processing tasks while preserving curvilinear structures. The experiments carried out on 2D, 3D, real, and synthetic images illustrate the relevance of the proposed gradient, and its use in variational frameworks for both denoising and segmentation tasks. [ABSTRACT FROM AUTHOR]

The study of eye gaze fixations on photographic images is an active research area. In contrast, the image sub-category of freehand sketches has not received as much attention for such studies. In this paper, we analyze the results of a free-viewing gaze fixation study conducted on 3904 freehand sketches distributed across 160 object categories. Our analysis shows that fixation sequences exhibit marked consistency within a sketch, across sketches of a category and even across suitably grouped sets of categories. This multi-level consistency is remarkable given the variability in depiction and extreme image content sparsity that characterizes hand-drawn object sketches. In this paper, we show that the multi-level consistency in the fixation data can be exploited to 1) predict a test sketch’s category given only its fixation sequence and 2) build a computational model which predicts part-labels underlying fixations on objects. We hope that our findings motivate the community to deem sketch-like representations worthy of gaze-based studies vis-a-vis photographic images. [ABSTRACT FROM PUBLISHER]

Hypergraph learning has been widely exploited in various image processing applications, due to its advantages in modeling the high-order information. Its efficacy highly depends on building an informative hypergraph structure to accurately and robustly formulate the underlying data correlation. However, the existing hypergraph learning methods are sensitive to non-Gaussian noise, which hurts the corresponding performance. In this paper, we present a noise-resistant hypergraph learning model, which provides superior robustness against various non-Gaussian noises. In particular, our model adopts low-rank representation to construct a hypergraph, which captures the globally linear data structure as well as preserving the grouping effect of highly correlated data. We further introduce a correntropy-induced local metric to measure the reconstruction errors, which is particularly robust to non-Gaussian noises. Finally, the Frobenious-norm-based regularization is proposed to combine with the low-rank regularizer, which enables our model to regularize the singular values of the coefficient matrix. By such, the non-zero coefficients are selected to generate a hyperedge set as well as the hyperedge weights. We have evaluated the proposed hypergraph model in the tasks of image clustering and semi-supervised image classification. Quantitatively, our scheme significantly enhances the performance of the state-of-the-art hypergraph models on several benchmark data sets. [ABSTRACT FROM AUTHOR]

Most variational formulations for structure-texture image decomposition force the structure images to have small norm in some functional spaces and to share a common notion of edges, i.e., large-gradients or large-intensity differences. However, such a definition makes it difficult to distinguish structure edges from oscillations that have fine spatial scale but high contrast. In this paper, we introduce a new model by learning deep variational priors for structure images without explicit training data. An alternating direction method of a multiplier algorithm and its modular structure are adopted to plug deep variational priors into an iterative smoothing process. The central observations are that convolution neural networks (CNNs) can replace the total variation prior, and are indeed powerful to capture the natures of structure and texture. We show that our learned priors using CNNs successfully differentiate high-amplitude details from structure edges, and avoid halo artifacts. Different from previous data-driven smoothing schemes, our formulation provides another degree of freedom to produce continuous smoothing effects. Experimental results demonstrate the effectiveness of our approach on various computational photography and image processing applications, including texture removal, detail manipulation, HDR tone-mapping, and non-photorealistic abstraction. [ABSTRACT FROM AUTHOR]

Multispectral imaging is of wide application for its capability in acquiring the spectral information of scenes. Due to hardware limitation, multispectral imaging device usually cannot achieve high-spatial resolution. To address the issue, this paper proposes a multispectral image super-resolution algorithm, referred as SRIF, by fusing the low-resolution multispectral image and the high-resolution (HR) RGB image. It deals with the general circumstance that image intensity is linear to scene radiance for multispectral imaging devices while is nonlinear and unknown for most RGB cameras. The SRIF algorithm first solves the inverse camera response function and a spectral sensitivity function of RGB camera, and establishes the linear relationship between multispectral and RGB images. Then the unknown HR multispectral image is efficiently reconstructed according to the linear image degradation models. Meanwhile, the edge structure of the reconstructed HR multispectral image is kept in accordance with that of the RGB image using a weighted total variation regularizer. The effectiveness of the SRIF algorithm is evaluated on both public datasets and our image set. Experimental results validate that the SRIF algorithm outperforms the state-of-the-arts in terms of both reconstruction accuracy and computational efficiency. [ABSTRACT FROM AUTHOR]

Image annotation aims to annotate a given image with a variable number of class labels corresponding to diverse visual concepts. In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept and 2) how to annotate an image with the optimal number of class labels. To address the first issue, we propose a novel multi-scale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Specifically, a novel two-branch deep neural network architecture is proposed, which comprises a very deep main network branch and a companion feature fusion network branch designed for fusing the multi-scale features computed from the main branch. The deep model is also made multi-modal by taking noisy user-provided tags as model input to complement the image input. For tackling the second issue, we introduce a label quantity prediction auxiliary task to the main label prediction task to explicitly estimate the optimal label number for a given image. Extensive experiments are carried out on two large-scale image annotation benchmark datasets, and the results show that our method significantly outperforms the state of the art. [ABSTRACT FROM AUTHOR]

Random walk is a popular and efficient algorithm for image segmentation, especially for extracting regions of interest (ROIs). One difficulty with the random walk algorithm is the requirement for solving a huge sparse linear system when applied to large images. Another limitation is its sensitivity to seeds distribution, i.e., the segmentation result depends on the number of seeds as well as their placement, which puts a burden on users. In this paper, we first propose a continuous random walk model with explicit coherence regularization (CRWCR) for the extracted ROI, which helps to reduce the seeds sensitivity, so as to reduce the user interactions. Then, a very efficient algorithm to solve the CRWCR model will be developed, which helps to remove the difficulty of solving huge linear systems. Our algorithm consists of two stages: initialization by performing one-dimensional random walk sweeping based on user-provided seeds, followed by the alternating direction scheme, i.e., Peaceman–Rachford scheme for further correction. The first stage aims to provide a good initial guess for the ROI, and it is very fast since we just solve a limited number of one-dimensional random walk problems. Then, this initial guess is evolved to the ideal solution by applying the second stage, which should also be very efficient since it fits well for GPU computing, and 10 iterations are usually sufficient for convergence. Numerical experiments are provided to validate the proposed model as well as the efficiency of the two-stage algorithm. [ABSTRACT FROM AUTHOR]