Search

Image generation from a single image using generative adversarial networks is quite interesting due to the realism of generated images. However, recent approaches need improvement for such realistic and diverse image generation, when the global context of the image is important such as in face, animal, and architectural image generation. This is mainly due to the use of fewer convolutional layers for mainly capturing the patch statistics and, thereby, not being able to capture global statistics very well. We solve this problem by using attention blocks at selected scales and feeding a random Gaussian blurred image to the discriminator for training. Our results are visually better than the state-of-the-art particularly in generating images that require global context. The diversity of our image generation, measured using the average standard deviation of pixels, is also better., Published in the Neurocomputing Journal

Semisupervised learning has been widely applied to deep generative model such as variational autoencoder. However, there are still limited work in noise-robust semisupervised deep generative model where the noise exists in both of the data and the labels simultaneously, which are referred to as outliers and noisy labels or compound noise. In this article, we propose a novel noise-robust semisupervised deep generative model by jointly tackling the noisy labels and outliers in a unified robust semisupervised variational autoencoder randomized generative adversarial network (URSVAE-GAN). Typically, we consider the uncertainty of the information of the input data in order to enhance the robustness of the variational encoder toward the noisy data in our unified robust semisupervised variational autoencoder (URSVAE). Subsequently, in order to alleviate the detrimental effects of noisy labels, a denoising layer is integrated naturally into the semisupervised variational autoencoder so that the variational inference is conditioned on the corrected labels. Moreover, to enhance the robustness of the variational inference in the presence of outliers, the robust β-divergence measure is employed to derive the novel variational lower bound, which already achieves competitive performance. This further motivates the development of URSVAE-GAN that collapses the decoder of URSVAE and the generator of a robust semisupervised generative adversarial network into one unit. By applying the end-to-end denoising scheme in the joint optimization, the experimental results demonstrate the superiority of the proposed framework by the evaluating on image classification and face recognition tasks and comparing with the state-of-the-art approaches.

In this paper, we propose an active contour model with a local variance force (LVF) term that can be applied to multi-phase image segmentation problems. With the LVF, the proposed model is very effective in the segmentation of images with noise. To solve this model efficiently, we represent the regularization term by characteristic functions and then design a minimization algorithm based on a modification of the iterative convolution-thresholding method (ICTM), namely ICTM-LVF. This minimization algorithm enjoys the energy-decaying property under some conditions and has highly efficient performance in the segmentation. To overcome the initialization issue of active contour models, we generalize the inhomogeneous graph Laplacian initialization method (IGLIM) to the multi-phase case and then apply it to give the initial contour of the ICTM-LVF solver. Numerical experiments are conducted on synthetic images and real images to demonstrate the capability of our initialization method, and the effectiveness of the local variance force for noise robustness in the multi-phase image segmentation.

Instead of being observed by human, multimedia data are now more and more fed into machines to perform different kinds of semantic analysis. One image may be analyzed multiple times by different machine vision algorithms for different purposes. While machine vision-oriented image compression has been studied, the existing methods are usually driven by a specific machine vision task, and may not be applicable for other tasks. We address the task-generic image compression, in the hope that an image is compressed once but used multiple times for different tasks, all with satisfactory performance. Our study is based on the end-to-end learned image compression. We focus ourselves on the distortion metric, i.e., finding out a task-agnostic metric to estimate the quality of reconstructed images. On the one hand, we study deep feature distance as the metric, which transforms images into a latent space by a pretrained convolutional network -- the latent space is believed to be more aligned to semantics -- and calculates distance in the latent space. On the other hand, inspired by the saliency mechanism, we study an importance-weighted pixel distance as the metric, where the weights are generated to reflect the importance of the pixels to semantics. Moreover, we combine the two distances into one metric to investigate their complementary nature. An extensive set of experiments are performed to evaluate these metrics. Experimental results show that the combined metric performs the best, and leads to 20.79%~42.69% bits saving under the same semantic analysis performance, compared to the same network but optimized for signal fidelity. Interestingly, we observe that using the combined metric also improves the visual quality of the reconstructed images.

Spike camera is a new type of bio-inspired vision sensor, each pixel of which perceives the brightness of the scene independently, and finally outputs 3-dimensional spatiotemporal spike streams. To bridge the spike camera and traditional frame-based vision, there is some works to reconstruct spike streams into regular images. However, the low spatial resolution (400×250) of the spike camera limits the quality of the reconstructed images. Thus, it is meaningful to explore a super-resolution reconstruction for spike streams. In this paper, we propose an end-to-end network to reconstruct high-resolution images from low-resolution spike streams. To utilize more spatiotemporal features of spike streams, our network adopts a multi-level features learning mechanism, including intra-stream feature extraction by spike encoder, inter-stream dependencies extraction based on optical flow module, and joint features learning via spike-based iterative projection. Experimental results demonstrate that our network is superior to the combination of state-of-the-art intensity image reconstruction methods and super-resolution networks on simulated and real datasets.

In this paper, we focus on learning optimal parameters for PDE-based image denoising and decomposition models. First, we learn the regularization parameter and the differential operator for gray-scale image denoising using the fractional Laplacian in combination with a bilevel optimization problem. In our setting the fractional Laplacian allows the use of Fourier transform, which enables the optimization of the denoising operator. We prove stable and explainable results as an advantage in comparison to machine learning approaches. The numerical experiments correlate with our theoretical model settings and show a reduction of computing time in contrast to the Rudin-Osher-Fatemi model. Second, we introduce a new regularized image decomposition model with the fractional Laplacian and the Riesz potential. We provide an explicit formula for the unique solution and the numerical experiments illustrate the efficiency.

Remote Sensing satellites acquire information about an area by analyzing the transmitted and reflected radio waves. The Earth Imaging satellites capture high resolution images which occupy more storage space on-board and consume a lot of bandwidth for downlink transmission. Multispectral sensors represent information of the images in multiple bands which are typically less than 15. The multispectral image compression algorithms aim to reduce the size of the images while preserving their quality. This paper involves a study of various algorithms used for compression of multispectral imagery.

Multi-camera networks are increasingly becoming pervasive in many monitoring and surveillance applications, and have attracted much attention in distributed systems with collaborative, real-time decision-making capabilities. While in-network data compression brings significant energy savings in camera nodes, signal representation using sparse approximations and overcomplete dictionaries have been shown to outperform traditional compression methods. In this work, an end-to-end and real-time solution is designed and implemented to enable energy-efficient and robust dictionary learning in distributed camera networks by leveraging the spatial correlation of the collected multimedia data. Traditional distributed dictionary learning relies on consensus-building algorithms, which involve communicating with neighboring nodes until convergence is achieved. Existing methods, however, do not exploit spatial correlations in camera networks for improved energy efficiency. In contrast, low-computational-complexity metrics are employed in this work to quantify and exploit the spatial correlation across camera nodes in a wireless network for efficient distributed dictionary learning and in-network image compression. The performance of the proposed approach is validated through extensive simulations on public datasets as well as via real-world experiments on a testbed composed of Raspberry Pi nodes.

Graph convolutional neural network (GCN) has effectively boosted the multi-label image recognition task by modeling correlation among labels. In previous methods, label correlation is computed based on statistical information through label diffusion, and therefore the same for all samples. This, however, makes graph inference on labels insufficient to handle huge variations among numerous image instances. In this paper, we propose an instance-aware graph convolutional neural network (IA_GCN) framework for the multi-label classification. As a whole, two fused branches of sub-networks are involved in the framework: a global branch modeling the whole image and a local branch exploring dependencies among regions of interests (ROIs). For both the branches, an image-dependent label correlation matrix (ID_LCM), fusing both the statistical LCM and an individual one of each image instance, is constructed to inject adaptive information of label-awareness into the learned features of the model through graph convolution. Specifically, the individual LCM of each image is obtained by mining the label dependencies based on the predicted label scores of those detected ROIs. In this process, considering the contribution differences of ROIs to multi-label classification, variational inference is introduced to learn adaptive scaling factors for those ROIs by considering their complex distribution. Finally, extensive experiments on MS-COCO and VOC datasets show that our proposed approach outperforms existing state-of-the-art methods.

Stitching images with parallax for naturalness remains a challenging problem. This paper proposes an image stitching method which preserves the flatness of planes in the scene for a natural look. Our method formulates the alignment of images as the camera parameters and the normal vectors of planes. Given a set of feature point matches, a process of grouping points into different layers and rejecting outliers is introduced. According to the epipolar constraint of corresponding points in two images, the focal length and the pose change of the camera are recovered simultaneously. Then, the normal vectors are estimated from the point pairs. To achieve good alignment and guide the warping of images, the model is combined with the mesh deformation as a global similarity constraint. In addition, bundle adjustment is adopted to maintain the consistency for stitching multiple images. Experiment shows that the proposed approach outperforms some state-of-the-art warps on real-world scenes.

The presence of noise is an intrinsic problem in acquisition processes for digital images. One way to enhance images is to combine the forward and backward diffusion equations. However, the latter problem is well known to be exponentially unstable with respect to any small perturbations on the final data. In this scenario, the final data can be regarded as a blurred image obtained from the forward process, and that image can be pixelated as a network. Therefore, we study in this work a regularization framework for the backward diffusion equation on graphs. Our aim is to construct a spectral graph-based solution based upon a cut-off projection. Stability and convergence results are provided together with some numerical experiments., 12 pages, 42 figures, 2 tables

In this paper, we consider the problem of descriptors construction for the task of content-based image retrieval using deep neural networks. The idea of neural codes, based on fully connected layers activations, is extended by incorporating the information contained in convolutional layers. It is known that the total number of neurons in the convolutional part of the network is large and the majority of them have little influence on the final classification decision. Therefore, in the paper we propose a novel algorithm that allows us to extract the most significant neuron activations and utilize this information to construct effective descriptors. The descriptors consisting of values taken from both the fully connected and convolutional layers perfectly represent the whole image content. The images retrieved using these descriptors match semantically very well to the query image, and also they are similar in other secondary image characteristics, like background, textures or color distribution. These features of the proposed descriptors are verified experimentally based on the IMAGENET1M dataset using the VGG16 neural network., Comment: 8

The general tensor-based methods can recover missing values of multidimensional images by exploiting the low-rankness on the pixel level. However, especially when considerable pixels of an image are missing, the low-rankness is not reliable on the pixel level, resulting in some details losing in their results, which hinders the performance of subsequent image applications (e.g., image recognition and segmentation). In this article, we suggest a novel multiscale feature (MSF) tensorization by exploiting the MSFs of multidimensional images, which not only helps to recover the missing values on a higher level, that is, the feature level but also benefits subsequent image applications. By exploiting the low-rankness of the resulting MSF tensor constructed by the new tensorization, we propose the convex and nonconvex MSF tensor train rank minimization (MSF-TT) to conjointly recover the MSF tensor and the corresponding original tensor in a unified framework. We develop the alternating directional method of multipliers (ADMMs) to solve the convex MSF-TT and the proximal alternating minimization (PAM) to solve the nonconvex MSF-TT. Moreover, we establish the theoretical guarantee of convergence for the PAM algorithm. Numerical examples of real-world multidimensional images show that the proposed MSF-TT outperforms other compared approaches in image recovery and the recovered MSF tensor can benefit the subsequent image recognition.

We introduce a general construction of many Hermitian LCD $[n, k]$ codes from a given Hermitian LCD $[n, k]$ code. Furthermore, we present some results on punctured codes and shortened codes of quaternary Hermitian LCD codes. As an application, we improve some of the previously known lower bounds on the largest minimum weights of quaternary Hermitian LCD codes of length $12 \le n \le 30$., 15 pages

Facing with rapidly increasing demands for analyzing high-order data or multiway data, feature-extracting methods become imperative for analysis and processing. The traditional feature-extracting methods, however, either need to overly vectorize the data and smash the original structure hidden in data, such as PCA and PCA-like methods, which is unfavorable to the data recovery, or cannot eliminate the redundant information very well, such as tucker decomposition (TD) and TD-like methods. To overcome these limitations, we propose a more flexible and more powerful tool, called the multiview principal components analysis (Multiview-PCA) in this article. By segmenting a random tensor into equal-sized subarrays called sections and maximizing variations caused by orthogonal projections of these sections, the Multiview-PCA finds principal components in a parsimonious and flexible way. In so doing, two new operations on tensors, the S -direction inner/outer product, are introduced to formulate tensor projection and recovery. With different segmentation ways characterized by section depth and direction, the Multiview-PCA can be implemented many times in different ways, which defines the sequential and global Multiview-PCA, respectively. These multiple Multiview-PCA take the PCA and PCA-like, and TD and TD-like as the special cases, which correspond to the deepest section depth and the shallowest section depth, respectively. We propose an adaptive depth and direction selection algorithm for the implementation of Multiview-PCA. The Multiview-PCA is then tested in terms of subspace recovery ability, compression ability, and feature extraction performance when applied to a set of artificial data, surveillance videos, and hyperspectral imaging data. All numerical results support the flexibility, effectiveness, and usefulness of Multiview-PCA.

In addition to 3D geometry, accurate representation of texture is important when digitizing real objects in virtual worlds. Based on a single consumer RGBD sensor, accurate texture representation for static objects can be realized by fusing multi-frame information; however, extending the process to dynamic objects, which typically have time-varying textures, is difficult. Thus, to address this problem, we propose a compact keyframe-based representation that decouples a dynamic texture into a basic static texture and a set of multiplicative changing maps. With this representation, the proposed method first aligns textures recorded from multiple keyframes with the reconstructed dynamic geometry of the object. Errors in the alignment and geometry are then compensated in an innovative iterative linear optimization framework. With the reconstructed texture, we then employ a scheme to synthesize the dynamic object from arbitrary viewpoints. By considering temporal and local pose similarities jointly, dynamic textures in all keyframes are fused to guarantee high-quality image generation. Experimental results demonstrate that the proposed method handles various dynamic objects, including faces, bodies, cloth, and toys. In addition, qualitative and quantitative comparisons demonstrate that the proposed method outperforms state-of-the-art solutions.

The field of artificial intelligence based image enhancement has been rapidly evolving over the last few years and is able to produce impressive results on non-astronomical images. In this work, we present the first application of Machine Learning based super-resolution (SR) and de-noising (DN) to enhance X-ray images from the European Space Agency’s XMM-Newton telescope. Using XMM-Newton images in band [0.5, 2] keV from the European Photon Imaging Camera pn detector (EPIC-pn), we develop XMM-SuperRes and XMM-DeNoise – deep learning-based models that can generate enhanced SR and DN images from real observations. The models are trained on realistic XMM-Newton simulations such that XMM-SuperRes will output images with two times smaller point-spread function and with improved noise characteristics. The XMM-DeNoise model is trained to produce images with 2.5× the input exposure time from 20 to 50 ks. When tested on real images, DN improves the image quality by 8.2 per cent, as quantified by the global peak-signal-to-noise ratio. These enhanced images allow identification of features that are otherwise hard or impossible to perceive in the original or in filtered/smoothed images with traditional methods. We demonstrate the feasibility of using our deep learning models to enhance XMM-Newton X-ray images to increase their scientific value in a way that could benefit the legacy of the XMM-Newton archive.

A finite element-based image segmentation strategy enhanced by an anisotropic mesh adaptation procedure is presented. The methodology relies on a split Bregman algorithm for the minimisation of a region-based energy functional and on an anisotropic recovery-based error estimate to drive mesh adaptation. More precisely, a Bayesian energy functional is considered to account for image spatial information, ensuring that the methodology is able to identify inhomogeneous spatial patterns in complex images. In addition, the anisotropic mesh adaptation guarantees a sharp detection of the interface between background and foreground of the image, with a reduced number of degrees of freedom. The resulting split-adapt Bregman algorithm is tested on a set of real images showing the accuracy and robustness of the method, even in the presence of Gaussian, salt and pepper and speckle noise., 41 pages, 13 figures, 5 tables

A multimode fiber represents the ultimate limit in miniaturization of imaging endoscopes. However, such a miniaturization usually comes as a cost of a low spatial resolution and a long acquisition time. Here we propose a fast superresolution-fiber-imaging technique employing compressive sensing through a multimode fiber with a data-driven machine-learning framework. We implement a generative adversarial network (GAN) to explore the sparsity inherent to the model and provide compressive reconstruction images that are not sparse in a representation basis. The proposed method outperforms other widespread compressive imaging algorithms in terms of both image quality and noise robustness. We experimentally demonstrate machine-learning ghost imaging below the diffraction limit at a sub-Nyquist speed through a thin multimode fiber probe. We believe that this work has great potential in applications in various fields ranging from biomedical imaging to remote sensing.

The normal growth of fishes is closely relevant to the density of mariculture. It is of great significance to accurately calculate the breeding area of specific sea area from satellite remote sensing images. However, there are no reports about cage segmentation and density detection based on remote sensing images so far. And the accurate segmentation of cages faces challenges from very large high-resolution images. Firstly, a new public mariculture cage data set is built. Secondly, the training set is augmented via sample variations to improve the robustness of the model. Then, for cage segmentation and density statistics, a new methodology based on Mask R-CNN is proposed. Using dividing and stitching technologies, the entire remote sensing test images of the cage can be accurately segmented. Finally, using the trained model, the object detection features and segmentation characteristics can be obtained at the same time. Considering only the area within the target detection frame, the proposed method can count the pixels in the segmented area, which can obtain accurate area and density while reducing time-consuming. Experimental results demonstrate that, compared with traditional contour extraction method and U-Net based scheme, the proposed scheme can significantly improve segmentation precision and model’s robustness. The relative error of the actual area is only 1.3%.

Affinity graph-based segmentation methods have become a major trend in computer vision. The performance of these methods relies on the constructed affinity graph, with particular emphasis on the neighborhood topology and pairwise affinities among superpixels. Due to the advantages of assimilating different graphs, a multi-scale fusion graph has a better performance than a single graph with single-scale. However, these methods ignore the noise from images which influences the accuracy of pairwise similarities. Multi-scale combinatorial grouping and graph fusion also generate a higher computational complexity. In this paper, we propose an adaptive fusion affinity graph (AFA-graph) with noise-free low-rank representation in an online manner for natural image segmentation. An input image is first over-segmented into superpixels at different scales and then filtered by the proposed improved kernel density estimation method. Moreover, we select global nodes of these superpixels on the basis of their subspace-preserving presentation, which reveals the feature distribution of superpixels exactly. To reduce time complexity while improving performance, a sparse representation of global nodes based on noise-free online low-rank representation is used to obtain a global graph at each scale. The global graph is finally used to update a local graph which is built upon all superpixels at each scale. Experimental results on the BSD300, BSD500, MSRC, SBD, and PASCAL VOC show the effectiveness of AFA-graph in comparison with state-of-the-art approaches., 12 pages, 12 figures

In this thesis, we develop various methods for the purpose of data denoising. We propose a method for Mean Square Error (MSE) estimation in Soft Thresholding. The MSE estimator is based on Minimum Noiseless Data Length (MNDL). Our simulation results show that this MSE estimate is a valuable comparison measure for different soft thresholding methods. Two denoising methods are proposed for analog domain: Mean Square Error EstiMation (MSEEM) which minimizes the worst case MSE estimate, and Noise Invalidation Denoising (NIDe) method which is based on the newly prosposed idea of noise signature. While MSEEM shown to be the optimum denoising method for non-sparse signals, NIDe approach outperforms the other well known denoising methods in presence of colored noise. In digital domain we address two interesting problems: 1) simultaneous denoising and quantization method, 2) denoising a digital signal in digital domain. For problem one, we propose a new method that generalizes the idea of dead zone estimation to a multi-level noise removal. An example of this method is shown for hyperspectral image denoising and compression. A digital domain denoising approach pioneers in answering the second problem with only one prior knowledge on the desired signal, that it is digital. The method provides the optimum reconstruction levels in the MSE sense. One of the critical steps of denoising process is the noise variance estimation. As a part of this thesis, we propose a novel noise variance estimation method for BayesShrink that outperforms conventional MAD-based noise variance estimation. Although BayesShrink is one of the most efficient denoising methods, no analytical analysis is available for it. Here, we study Bayes estimators for General Gaussian Distribued (GGD) data and provide the theoretical justification for BayesShrink. This study enables us to generalize the BayesShrink threshold to Generalized BayesShrink which outperforms the BayesShrink itself.

This paper presents an unsupervised and semi-automatic image segmentation approach where we formulate the segmentation as an inference problem based on unary and pairwise assignment probabilities computed using low-level image cues. The inference is solved via a probabilistic graph matching scheme, which allows rigorous incorporation of low-level image cues and automatic tuning of parameters. The proposed scheme is experimentally shown to compare favorably with contemporary semi-supervised and unsupervised image segmentation schemes, when applied to contemporary state-of-the-art image sets.

The categorical distribution is a natural representation of uncertainty in multi-class segmentations. In the two-class case the categorical distribution reduces to the Bernoulli distribution, for which grayscale morphology provides a range of useful operations. In the general case, applying morphological operations on uncertain multi-class segmentations is not straightforward as an image of categorical distributions is not a complete lattice. Although morphology on color images has received wide attention, this is not so for color-coded or categorical images and even less so for images of categorical distributions. In this work, we establish a set of requirements for morphology on categorical distributions by combining classic morphology with a probabilistic view. We then define operators respecting these requirements, introduce protected operations on categorical distributions and illustrate the utility of these operators on two example tasks: modeling annotator bias in brain tumor segmentations and segmenting vesicle instances from the predictions of a multi-class U-Net., Comment: Major rewrite: added thorough review and comparison of related work. Added extra method. Cleanup of proofs. Clearer examples. More figures. Submitted to JMIV

We introduce the group-equivariant autoencoder (GE-autoencoder) -- a deep neural network (DNN) method that locates phase boundaries by determining which symmetries of the Hamiltonian have spontaneously broken at each temperature. We use group theory to deduce which symmetries of the system remain intact in all phases, and then use this information to constrain the parameters of the GE-autoencoder such that the encoder learns an order parameter invariant to these ``never-broken'' symmetries. This procedure produces a dramatic reduction in the number of free parameters such that the GE-autoencoder size is independent of the system size. We include symmetry regularization terms in the loss function of the GE-autoencoder so that the learned order parameter is also equivariant to the remaining symmetries of the system. By examining the group representation by which the learned order parameter transforms, we are then able to extract information about the associated spontaneous symmetry breaking. We test the GE-autoencoder on the 2D classical ferromagnetic and antiferromagnetic Ising models, finding that the GE-autoencoder (1) accurately determines which symmetries have spontaneously broken at each temperature; (2) estimates the critical temperature in the thermodynamic limit with greater accuracy, robustness, and time-efficiency than a symmetry-agnostic baseline autoencoder; and (3) detects the presence of an external symmetry-breaking magnetic field with greater sensitivity than the baseline method. Finally, we describe various key implementation details, including a new method for extracting the critical temperature estimate from trained autoencoders and calculations of the DNN initialization and learning rate settings required for fair model comparisons., Comment: 28 pages, 14 figures. For associated code repository see https://github.com/dagrawa2/ssb_detection_ising

The localized nature of curvelet functions, together with their frequency and dip characteristics, makes the curvelet transform an excellent choice for processing seismic data. In this work, a denoising method is proposed based on a combination of the curvelet transform and a whitening filter along with procedure for noise variance estimation. The whitening filter is added to get the best performance of the curvelet transform under coherent and incoherent correlated noise cases, and furthermore, it simplifies the noise estimation method and makes it easy to use the standard threshold methodology without digging into the curvelet domain. The proposed method is tested on pseudo-synthetic data by adding noise to real noise-less data set of the Netherlands offshore F3 block and on the field data set from east Texas, USA, containing ground roll noise. Our experimental results show that the proposed algorithm can achieve the best results under all types of noises (incoherent or uncorrelated or random, and coherent noise).

We perform imaging of a fundamental string from string scattering amplitudes, and show that its image is a double slit., Comment: 19 pages, 10 figures

Spectral unmixing (SU) of hyperspectral images (HSIs) is one of the important areas in remote sensing (RS) that needs to be carefully addressed in different RS applications. Despite the high spectral resolution of the hyperspectral data, the relatively low spatial resolution of the sensors may lead to mixture of different pure materials within the image pixels. In this case, the spectrum of a given pixel recorded by the sensor can be a combination of multiple spectra each belonging to a unique material in that pixel. Spectral unmixing is then used as a technique to extract the spectral characteristics of the different materials within the mixed pixels and to recover the spectrum of each pure spectral signature, called endmember. Block-sparsity exists in hyperspectral images as a result of spectral similarity between neighboring pixels. In block-sparse signals, the nonzero samples occur in clusters and the pattern of the clusters is often supposed to be unavailable as prior information. This paper presents an innovative spectral unmixing approach for HSIs based on block-sparse structure. Hyperspectral unmixing problem is solved using pattern coupled sparse Bayesian learning strategy (PCSBL). To evaluate the performance of the proposed SU algorithm, it is tested on both synthetic and real hyperspectral data and the quantitative results are compared to those of other state-of-the-art methods in terms of abundance angle distance and mean squared error. The achieved results show the superiority of the proposed algorithm over the other competing methods by a significant margin., 25 pages, 8 figures, 2 tables, accepted for publication in journal

Although spatial information of images usually enhance the robustness of the Fuzzy C-Means (FCM) algorithm, it greatly increases the computational costs for image segmentation. To achieve a sound trade-off between the segmentation performance and the speed of clustering, we come up with a Kullback-Leibler (KL) divergence-based FCM algorithm by incorporating a tight wavelet frame transform and a morphological reconstruction operation. To enhance FCM's robustness, an observed image is first filtered by using the morphological reconstruction. A tight wavelet frame system is employed to decompose the observed and filtered images so as to form their feature sets. Considering these feature sets as data of clustering, an modified FCM algorithm is proposed, which introduces a KL divergence term in the partition matrix into its objective function. The KL divergence term aims to make membership degrees of each image pixel closer to those of its neighbors, which brings that the membership partition becomes more suitable and the parameter setting of FCM becomes simplified. On the basis of the obtained partition matrix and prototypes, the segmented feature set is reconstructed by minimizing the inverse process of the modified objective function. To modify abnormal features produced in the reconstruction process, each reconstructed feature is reassigned to the closest prototype. As a result, the segmentation accuracy of KL divergence-based FCM is further improved. What's more, the segmented image is reconstructed by using a tight wavelet frame reconstruction operation. Finally, supporting experiments coping with synthetic, medical and color images are reported. Experimental results exhibit that the proposed algorithm works well and comes with better segmentation performance than other comparative algorithms. Moreover, the proposed algorithm requires less time than most of the FCM-related algorithms., Comment: This paper has been withdrawn by the author due to a crucial definition error of objective function

Edge Detection is the first stage in the image division into separate parts. Image division is the partitioning of a digital image to the different zones or the set of pixels. Edge detection is one of the techniques applied in digital image processing often. The purpose of detecting pixels is to match the edges in the image. Filtering, Enhancement, and Detection are three steps in edge detection. Images are usually destroyed by casual changes in intensity intervals called noise or confusion. some noise variations include salt and pepper, pulse, and Gaussian. However, there is a relation between edge detection power and noise reduction. Using filters to the noise reduction causes the loss of edge detection power. For facilitating the edges detection, it is essential for the determination of pixels’ intensity constraints in their neighborhood. Many points in an image have a nontransparent slope, and all of them are not the edges of the joint space. Therefore, some of the linear and nonlinear methods such as Sobel, Prewitt, and Robert have to be used to determine the edge points. The fuzzy logic and the system based on it, is one of the most effective methods for edge detection. This paper presents an optimized rule-based fuzzy inference system and designs the efficiency mask matric. The simulation results for edge detection are presented using the traditional edge detection techniques, including Binary Filter, Sobel Filter, Prewitt Filter, and Robert Filter. Also, it is presented using the fuzzy approach. The simulation results show that the designed fuzzy system has been able to detect the edges of the image more accurately and help to increase the sharpness and quality of the edges. Therefore, the proposed method has more accurate and more reliable results and reduces false edge detection comparison to the traditional methods.

Deep unsupervised learning extracts meaningful features from unlabeled images and simultaneously serves downstream tasks in computer vision. The basic process of deep clustering methods can include features learning and clustering assignment. To enhance the discriminative ability of the features and further improve the clustering performances, a new deep clustering method namely ACMEC (asymmetric convolutional denoising autoencoder with manifold spatial embedding clustering) is proposed. In this method, an asymmetric convolution denoising autoencoder is employed to extract visual features from images, and a manifold learning algorithm is used to obtain more distinctive features, followed by a Gaussian Mixture Model (GMM) is for clustering learning. The stability of feature space is guaranteed using separately training mechanism. In addition, reconstruction from noisy images enhances the robustness of feature networks. Experimental results on nine benchmark datasets demonstrate that the proposed ACMEC method can provide the better performances such as 0.979 clustering accuracy on the MNIST dataset and 0.668 on the fashion-MNIST dataset. ACMEC is a comparable competitor to the N2D (not too deep clustering) algorithm that is with 0.979 and 0.672 clustering accuracies respectively. Moreover, it is 16.1% higher than DEC algorithm on the fashion-MNIST dataset.

Convolutional neural network (CNN) algorithm is a very important branch of deep learning research, which has been widely applied in many fields and achieved excellent results, especially in computer vision, where convolutional neural network has made breakthroughs in image classification and object detection. Convolutional neural network architecture can realize more efficient network training through the final combination of different modules, and the convolutional neural network training does not need to actively extract image features and can directly carry out end-to-end training and prediction. At first, this paper analyzed some problems of the current image recognition and expounds the progress of convolution neural network in image recognition and then studied the traditional algorithm of target recognition, including traditional recognition algorithm framework of target, the target orientation, feature extraction, classifier classification, etc., and the traditional target recognition algorithm is compared with those of the target recognition algorithm of deep learning. On the basis of the above research, an improved model of CNN is proposed, which focuses on the structural design and network optimization of convolutional neural network and designs a more efficient convolutional neural network. Test experiments verify the effectiveness of the proposed model, which not only achieves lower error rate, but also greatly reduces the number of network parameters and has stronger learning ability.

In order to solve the problem of noise sensitivity and low precision in image segmentation, a new image segmentation method based on super-pixel with neighborhood constrained and multi-level FCM clustering is proposed. Firstly, an improved SLIC algorithm of FCM is designed based on the local pixel region of the image to get the refined super-pixel. Then, the improved FCM clustering algorithm is combined with the spatial relationship of neighborhood super-pixel to mark the super-pixel categories. In order to verify the effectiveness of the algorithm, the artificial synthetic image with noise and natural image without noise are used for comparison experiments. The experimental result show the algorithm proposed in this paper can not only suppress the noise, but also improve the processing accuracy.

This paper proposes the study of motion video image classification and recognition, extracts the motion target image features, designs the image classification process, and establishes the neural network image classification model to complete the image recognition. In view of the different angles of the same element, the motion video image classification and recognition under the neural network are completed by using the error back-propagation algorithm. The performance of the proposed method is verified by simulation experiments. Experimental results show that the proposed method has a high recognition rate of moving video images, the accuracy rate is more than 98%, and the image recognition classification is comprehensive. The proposed method can classify the elements in the motion video image, which solves the technical problem that the traditional method cannot identify unclear images and has low recognition accuracy.

This work presents a novel dense RGB-D SLAM approach for dynamic planar environments that enables simultaneous multi-object tracking, camera localisation and background reconstruction. Previous dynamic SLAM methods either rely on semantic segmentation to directly detect dynamic objects; or assume that dynamic objects occupy a smaller proportion of the camera view than the static background and can, therefore, be removed as outliers. Our approach, however, enables dense SLAM when the camera view is largely occluded by multiple dynamic objects with the aid of camera motion prior. The dynamic planar objects are separated by their different rigid motions and tracked independently. The remaining dynamic non-planar areas are removed as outliers and not mapped into the background. The evaluation demonstrates that our approach outperforms the state-of-the-art methods in terms of localisation, mapping, dynamic segmentation and object tracking. We also demonstrate its robustness to large drift in the camera motion prior., Comment: 8 papges, 9 figures

Deep convolutional neural networks have achieved state-of-the-art performance for the removal of atmospheric obscuration. However, most relevant studies have focused on eliminating the effects of atmospheric obscuration but not on the glare in images caused by reflected sunlight. On the basis of a glare image formation model, we propose a deep trident decomposition network with a large-scale sun glare image dataset for glare removal from single images. Specifically, the proposed network is designed and implemented with a trident decomposition module for decomposing an input glare image into occlusion, foreground, and coarse glare-free images by exploring background features from spatial locations. Moreover, a residual refinement module is adopted to refine the coarse glare-free image into fine glare-free image by learning the residuals from features of multiscale receptive field. The experimental results indicated that the proposed network significantly outperforms state-of-the-art atmospheric obscuration removal networks on the built dataset.

Context Prostate-specific membrane antigen (PSMA) positron emission tomography/magnetic resonance imaging (PET/MRI) is a novel imaging technique with several potential applications in the prostate cancer (PCa) setting. Objective To perform a systematic review of the current evidence regarding the diagnostic performance of PSMA PET/MRI in patients with primary and recurrent PCa. Evidence acquisition A comprehensive bibliographic search on the MEDLINE and Cochrane Library databases was performed in October 2020. The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement. Studies were deemed eligible if they assessed patients with primary or recurrent PCa (P) undergoing PSMA PET/MRI (I) with or without comparison with other imaging techniques (C) in order to evaluate its diagnostic performance (O). Retrospective and prospective primary clinical studies were included. Results of previous meta-analyses were reported. Evidence synthesis A total of 23 original articles and three meta-analyses were included. Limited evidence on PSMA PET/MRI is available, especially in the setting of partial gland ablation. PET/MRI can be an effective imaging modality for detecting primary PCa, showing higher accuracy than multiparametric MRI alone. It provides accurate local staging of primary PCa; however, there are contradictory results in this context when its performance is compared with other imaging techniques. PET/MRI also shows high performance for restaging and detecting tumor recurrence, even at low prostate-specific antigen levels. Conclusions PSMA PET/MRI could represent a valuable tool in the management of patients with primary and recurrent PCa. No specific recommendations can be provided. Patient summary Encouraging data regarding the benefits of prostate-specific membrane antigen positron emission tomography/magnetic resonance imaging in patients with prostate cancer are emerging from the literature.

Aiming at the problem of low efficiency and insufficient accuracy of threshold solution in multithreshold sculpture image segmentation, this paper proposes a threshold segmentation algorithm for sculpture images based on sparse decomposition. In this paper, sparse decomposition is introduced to optimize the model to reduce the impact of local noise on segmentation accuracy, and an energy functional based on pixel coconstraint is built to make up for the defect that pixels cannot retain local details. At the same time, the weighted sum of elite solution sets is used to determine Neighborhood centers increase communication between groups. Experiments show that compared with other algorithms, the above method has significant advantages in convergence efficiency and accuracy.

The emergence of portable devices provides great convenience for image acquisition, but the image resolution is too low, which affects the identification and use of the image. Multiframe super-resolution algorithm makes a great contribution to extracting image features, but there is a problem with too much computation. Based on this, this paper proposes a low-resolution image fusion algorithm based on deep learning. Based on the introduction of extracting image features from low-resolution images, this paper proposes to fuse multiple low-resolution images, reduce the time of calculation error per frame, reduce the amount of calculation, extract feature information, and improve the alexnet network model to realize image extraction. The convolutional self-coding fusion network is used to realize image fusion and complete the reconstruction and fusion of low-resolution images. In the simulation analysis of the fusion algorithm, objective indexes such as network structure and loss function are selected to evaluate the effectiveness of the algorithm. This paper reduces the computational complexity of image acquisition and ensures that the amount of calculation can be reduced in each iteration. Analyze the effectiveness of the fusion algorithm, comprehensively evaluate the objective indicators, and verify the advantages of the algorithm through indicators such as gradient change and information entropy.

Restoration of blurred images corrupted with impulse noise has attracted increasing attention due to its great potential toward practical applications. To guarantee the noisy image deblurring performance, total variation (TV), one of the most popular regularizers, has been extensively exploited in the literature. This local regularizer is known for its capability of preserving edges and penalizing oscillations but not sharp discontinuities. However, TV regularized variational models commonly suffer from undesirable staircase-like artifacts in homogeneous regions. The reason behind this phenomenon is that TV regularizer favors solutions that are piecewise constant in numerical experiments. To overcome the model-dependent deficiency, we proposed to replace the commonly used local regularizer by its nonlocal extension, i.e., nonlocal total variation (NLTV) regularizer. By taking advantage of the high degree of self-similarity within images, the proposed nonlocal variational model is able to remove both blurring and impulse noise while preserving meaningful image details. To guarantee solution stability and efficiency, the resulting nonsmooth image restoration problem was effectively dealt with using an alternating optimization algorithm. In addition, to further enhance the image quality, a local variance estimator-based automatic method was introduced to calculate the spatially adapted regularization parameters during image restoration. Extensive experiments have demonstrated that the proposed method could achieve superior imaging performance compared with several state-of-the-art image restoration methods.

The image denoising model based on convolutional neural network (CNN) can achieve a good denoising effect. However, its robustness is poor, and it is not suitable for direct noise removal tasks. Differently, the image denoising method based on the diffusion equation is more stable and has theoretical guarantees. In order to give full play to the advantages of CNN and diffusion equation in image denoising, this paper proposes a speckle noise denoising model via a combination of the two tools. Firstly, based on the mathematical model of speckle noise, a class of neural network speckle noise removal model which mixes residual learning and structure learning is proposed using image decomposition theory. Then, in order to solve the hyperparameter problem that the model depends on noise variance, a noise variance estimation algorithm based on a nonlinear diffusion equation is proposed. Finally, a speckle noise denoising model based on diffusion equation and CNN is obtained. Numerical simulation experiments verify the accuracy of the variance estimation algorithm and also the denoising effect and practical application value of the proposed method.

The detection of image edges plays an important role for image processing. In view of the fact that these existing methods cannot effectively detect the edge of the image when facing the image with rich details. This paper proposes a novel method of asymmetric spike-timing-dependent plasticity (STDP) image edge detection based on the visual physiological mechanism. In the proposed method, the original image is preprocessed by the Gabor filter to simulate the visual physiological orientation characteristics to obtain the image information in different directions, and the orientation feature fusion is used to reconstruct the primary edge feature information of the image. Then, based on the mechanism of the visual nervous system, a neuron network composed of dynamic synapses based on the asymmetric STDP mechanism is constructed to further process it to obtain impulse response images. In order to eliminate disturbance of the neuron’s system noise on the impulse response image, the impulse response image is filtered by a Gaussian filter. Then, the lateral inhibition between neurons is applied to refine the filtered image edges. Finally, the result is normalized, and the final edge of the experimental image is obtained. Experimental results based on the colony image data set collected in the laboratory indicate that the proposed method achieved better performance than these state-of-the-art methods; meanwhile, the AUC value remains above 0.6.

Multimodal tasks based on attention mechanism and language face numerous problems. Based on multimodal hierarchical attention mechanism and genetic neural network, this paper studies the application of image segmentation algorithm in data completion and 3D scene reconstruction. The algorithm refers to the process of concentrating attention that humans subjectively pay attention to and calculates the difference between each pixel in the genetic neural network test image in the color space and the average value of the target image, which solves the problem of static feature maps and dynamic feature maps of image sequences. In addition, in view of the problem that the number of attention enhancement feature extraction modules is too large and the parameters are too large, the recursive mechanism is used as the feature extraction branch, and new model parameters are not added when the network depth is increased. The simulation results show that the accuracy of the improved image saliency detection algorithm based on the attention mechanism reaches 89.7%, and the difference between the average value of the single-point pixel and the target image is reduced to 0.132, which further promotes the practicability and reliability of the image segmentation model.

Many application domains, spanning from computational photography to medical imaging, require recovery of high-fidelity images from noisy, incomplete or partial/compressed measurements. State-of-the-art methods for solving these inverse problems combine deep learning with iterative model-based solvers, a concept known as deep algorithm unfolding or unrolling. By combining a-priori knowledge of the forward measurement model with learned proximal image-to-image mappings based on deep networks, these methods yield solutions that are both physically feasible (data-consistent) and perceptually plausible (consistent with prior belief). However, current proximal mappings based on (predominantly convolutional) neural networks only implicitly learn such image priors. In this paper, we propose to make these image priors fully explicit by embedding deep generative models in the form of normalizing flows within the unfolded proximal gradient algorithm, and training the entire algorithm end-to-end for a given task. We demonstrate that the proposed method outperforms competitive baselines on various image recovery tasks, spanning from image denoising to inpainting and deblurring, effectively adapting the prior to the restoration task at hand

Image denoising is a vital image processing phase aiming to improve the quality of images and to make them more informative. In this paper, we propose a blind denoising approach for removing the outliers (impulsive disturbances) from digital images, by combining the random sample consensus (RANSAC) and compressive sensing (CS) principles. The proposed approach exploits the fact that images are highly concentrated in the domain of two-dimensional discrete cosine transform (2D- DCT). The sparsity (high concentration) in the transform domain is used in both detection and reconstruction of pixels affected by high disturbances. The image pixels not affected by the noise are found using the RANSAC-based methodology and they are further used as available measurements in the CS reconstruction. The affected pixels are considered unavailable and they are recovered by the CS procedure. The presented approach does not require any disturbance-related assumptions regarding the statistical behavior of the noise or about the range of its values. The theory is verified on examples with 55 images. The comparative analysis against several state-of-the-art methods, done with full-reference and no-reference quality metrics, suggests that the proposed method can be used as an efficient tool for image denoising.

Based on a nonlocal Laplacian operator, a novel edge detection method of the grayscale image is proposed in this paper. This operator utilizes the information of neighbor pixels for a given pixel to obtain effective and delicate edge detection. The nonlocal edge detection method is used as an initialization for solving the Allen-Cahn equation to achieve two-phase segmentation of the grayscale image. Efficient exponential time differencing (ETD) solvers are employed in the time integration, and finite difference method is adopted in space discretization. The maximum bound principle and energy stability of the proposed numerical schemes are proved. The capability of our segmentation method has been verified in numerical experiments for different types of grayscale images.

The Chan-Vese (CV) model is a classic region-based method in image segmentation. However, its piecewise constant assumption does not always hold for practical applications. Many improvements have been proposed but the issue is still far from well solved. In this work, we propose an unsupervised image segmentation approach that integrates the CV model with deep neural networks, which significantly improves the original CV model's segmentation accuracy. Our basic idea is to apply a deep neural network that maps the image into a latent space to alleviate the violation of the piecewise constant assumption in image space. We formulate this idea under the classic Bayesian framework by approximating the likelihood with an evidence lower bound (ELBO) term while keeping the prior term in the CV model. Thus, our model only needs the input image itself and does not require pre-training from external datasets. Moreover, we extend the idea to multi-phase case and dataset based unsupervised image segmentation. Extensive experiments validate the effectiveness of our model and show that the proposed method is noticeably better than other unsupervised segmentation approaches.

Cross-spectral recognition is still an open challenge in iris recognition. In cross-spectral iris recognition, there exist distinct device-specific bands between near-infrared (NIR) and visible (VIS) images, resulting in the distribution gap between samples from different spectra and thus severe degradation in recognition performance. To tackle this problem, we propose a new cross-spectral iris recognition method to learn spectral-invariant features by estimating device-specific bands. In the proposed method, Gabor Trident Network (GTN) first utilizes the Gabor function’s priors to perceive iris textures under different spectra, and then codes the device-specific band as the residual component to assist the generation of spectral-invariant features. By investigating the device-specific band, GTN effectively reduces the impact of device-specific bands on identity features. Besides, we make three efforts to further reduce the distribution gap. First, Spectral Adversarial Network (SAN) adopts a class-level adversarial strategy to align feature distributions. Second, Sample-Anchor (SA) loss upgrades triplet loss by pulling samples to their class center and pushing away from other class centers. Third, we develop a higher-order alignment loss to measures the distribution gap according to space bases and distribution shapes. Extensive experiments on five iris datasets demonstrate the efficacy of our proposed method for cross-spectral iris recognition.

Purpose: Proton-induced secondary-electron-bremsstrahlung (SEB) imaging is a promising method for estimating the ranges of particle beam. However, SEB images do not directly represent dose distributions of particle beams. In addition, the ranges estimated from measured images were deviated because of limited spatial resolutions of the developed x-ray camera as well as statistical noise in the images. To solve these problems, we proposed a method for predicting high-resolution dose images from SEB images with various count level using a deep learning (DL) approach for range and width verification. Methods: In this study, we adopted the double U-Net model, which is a previously proposed deep convolutional network model. The first U-Net model in the double U-Net model was used to denoise the SEB images with various count level. The first U-Net model for denoising was trained on 8000 pairs of SEB images with various count level and noise-free images which were created by a sophisticated in-house developed model function. The second U-Net model for dose prediction was trained using 8000 pairs of denoised SEB images from the first U-Net model and high-resolution dose images generated by Monte Carlo simulation. Results: For both simulation and measurement data, the trained DL model could successfully predict high-resolution dose images which showed a clear Bragg peak and no statistical noise. The difference of the range and width was less than 2.1 mm, even from the SEB images measured with a decrease in the number of irradiated protons to less than 11% of 3.2 ×10^11 protons. Conclusions: High-resolution dose images from measured and simulated SEB images were successfully predicted by using the trained DL model for protons. Our proposed DL model was feasible to predict dose images accurately even with smaller number of irradiated protons.

We propose a ParametRIc MAnifold Learning (PRIMAL) algorithm for Gaussian mixtures models (GMM), assuming that GMMs lie on or near to a manifold of probability distributions that is generated from a low-dimensional hierarchical latent space through parametric mappings. Inspired by principal component analysis (PCA), the generative processes for priors, means and covariance matrices are modeled by their respective latent space and parametric mapping. Then, the dependencies between latent spaces are captured by a hierarchical latent space by a linear or kernelized mapping. The function parameters and hierarchical latent space are learned by minimizing the reconstruction error between ground-truth GMMs and manifold-generated GMMs, measured by Kullback-Leibler Divergence (KLD). Variational approximation is employed to handle the intractable KLD between GMMs and a variational EM algorithm is derived to optimize the objective function. Experiments on synthetic data, flow cytometry analysis, eye-fixation analysis and topic models show that PRIMAL learns a continuous and interpretable manifold of GMM distributions and achieves a minimum reconstruction error.