Search

We feature papers on recent developments in the areas of adaptive signal processing, machine learning, and deep learning using information theory and entropy to improve performance in widespread and popular problems, and also to provide effective solutions to emerging problems. Our goal is to publish recent developments in the areas of adaptive signal processing, machine learning, and deep learning using information theory and entropy to improve performance in widespread and popular problems, and also to provide effective solutions to emerging problems. This Special Issue on "Adaptive Signal Processing and Machine Learning Using Entropy and Information Theory" was birthed from observations of the recent trend in the literature. Entropy-based cost functions have replaced mean-square-error (MSE)-based ones and have been widely used in adaptive signal processing and machine learning to improve performance by designing and optimizing effective and specific models that fit the data, even in noisy and adverse conditions. [Extracted from the article]

Recent studies on watermarking techniques based on image carriers have demonstrated new approaches that combine adversarial perturbations against steganalysis with embedding distortions. However, while these methods successfully counter convolutional neural network-based steganalysis, they do not adequately protect the data of the carrier itself. Recognizing the high sensitivity of Deep Neural Networks (DNNs) to small perturbations, we propose HAG-NET, a method based on image carriers, which is jointly trained by the encoder, decoder, and attacker. In this paper, the encoder generates Adversarial Steganographic Examples (ASEs) that are adversarial to the target classification network, thereby providing protection for the carrier data. Additionally, the decoder can recover secret data from ASEs. The experimental results demonstrate that ASEs produced by HAG-NET achieve an average success rate of over 99% on both the MNIST and CIFAR-10 datasets. ASEs generated with the attacker exhibit greater robustness in terms of attack ability, with an average increase of about 3.32%. Furthermore, our method, when compared with other generative stego examples under similar perturbation strength, contains significantly more information according to image information entropy measurements. [ABSTRACT FROM AUTHOR]

To further reduce the size of the neural network model and enable the network to be deployed on mobile devices, a novel fusion pruning algorithm based on information entropy stratification is proposed in this paper. Firstly, the method finds similar filters and removes redundant parts by Affinity Propagation Clustering, then secondly further prunes the channels by using information entropy stratification and batch normalization (BN) layer scaling factor, and finally restores the accuracy training by fine-tuning to achieve a reduced network model size without losing network accuracy. Experiments are conducted on the vgg16 and Resnet56 network using the cifar10 dataset. On vgg16, the results show that, compared with the original model, the parametric amount of the algorithm proposed in this paper is reduced by 90.69% and the computation is reduced to 24.46% of the original one. In ResNet56, we achieve a 63.82%-FLOPs reduction by removing 63.53% parameters. The memory occupation and computation speed of the new model are better than the baseline model while maintaining a high network accuracy. Compared with similar algorithms, the algorithm has obvious advantages in the dimensions of computational speed and model size. The pruned model is also deployed to the Internet of Things (IoT) as a target detection system. In addition, experiments show that the proposed model is able to detect targets accurately with low reasoning time and memory. It takes only 252.84 ms on embedded devices, thus matching the limited resources of IoT. [ABSTRACT FROM AUTHOR]

Feature recognition and fault diagnosis plays an important role in equipment safety and stable operation of rotating machinery. In order to cope with the complexity problem of the vibration signal of rotating machinery, a feature fusion model based on information entropy and probabilistic neural network is proposed in this paper. The new method first uses information entropy theory to extract three kinds of characteristics entropy in vibration signals, namely, singular spectrum entropy, power spectrum entropy, and approximate entropy. Then the feature fusion model is constructed to classify and diagnose the fault signals. The proposed approach can combine comprehensive information from different aspects and is more sensitive to the fault features. The experimental results on simulated fault signals verified better performances of our proposed approach. In real two-span rotor data, the fault detection accuracy of the new method is more than 10% higher compared with the methods using three kinds of information entropy separately. The new approach is proved to be an effective fault recognition method for rotating machinery. [ABSTRACT FROM AUTHOR]

Designing a high-performance neural network is a difficult task. Neural architecture search (NAS) methods aim to solve this process. However, the construction of a high-quality accuracy predictor, which is a key component of NAS, usually requires significant computation. Therefore, zero-shot proxy-based NAS methods have been actively and extensively investigated. In this work, we propose a new efficient zero-shot proxy, Incorporated-Score, to rank deep neural network architectures instead of using an accuracy predictor. The proposed Incorporated-Score proxy is generated by incorporating the zen-score and entropy information of the network, and it does not need to train any network. We then introduce an optimal NAS algorithm called Incorporated-NAS that targets the maximization of the Incorporated-Score of the neural network within the specified inference budgets. The experiments show that the network designed by Incorporated-NAS with Incorporated-Score outperforms the previously proposed Zen-NAS and achieves a new SOTAaccuracy on the CIFAR-10, CIFAR-100, and ImageNet datasets with a lightweight scale. [ABSTRACT FROM AUTHOR]

The air traffic control sector (ATCS) is the basic unit of the airspace system. If we can identify the congestion of an ATCS, it will help provide decision support for planning and daily operations. However, current methods mainly characterize congestion from the static structure and the dynamic operational features, resulting in poor generalization and operability. To this end, we propose a deep learning method from the perspective of complex networks. It takes aircraft as nodes to construct an aircraft network and utilizes the complexity indices to characterize it. So, the problem of identifying congestion becomes the complexity of the aircraft network. Inspired by active learning methods, we construct a deep active learning (DAL) model for congestion recognition. It adopts an iterative semi-supervised approach to reduce the number of labeled samples while ensuring recognition performance. To make full use of a large number of unlabeled samples, the sparse autoencoder is employed to characterize all labeled samples and unlabeled samples. The hidden layer of the deep neural network is constructed by stacking. In the process of active learning iteration, minimum confidence, marginal sampling, and information entropy are introduced as measures to select samples from the unlabeled sample set with significantly different features from the labeled sample set. The model is applied to three representative sectors in China's airspace as cases. Results suggest that DAL can reduce the labeled sample set's redundancy and achieve the desired performance with the smallest number of samples. Additionally, DAL is superior to the existing mainstream methods in the four objective evaluation indices. [ABSTRACT FROM AUTHOR]

This paper studies contrastive divergence (CD) learning algorithm and proposes a new algorithm for training restricted Boltzmann machines (RBMs). We derive that CD is a biased estimator of the log-likelihood gradient method and make an analysis of the bias. Meanwhile, we propose a new learning algorithm called average contrastive divergence (ACD) for training RBMs. It is an improved CD algorithm, and it is different from the traditional CD algorithm. Finally, we obtain some experimental results. The results show that the new algorithm is a better approximation of the log-likelihood gradient method and outperforms the traditional CD algorithm. [ABSTRACT FROM AUTHOR]

We propose a numerical method to learn maximum entropy (MaxEnt) distributions with spatio-temporal constraints from experimental spike trains. This is an extension of two papers, [10] and [4], which proposed the estimation of parameters where only spatial constraints were taken into account. The extension we propose allows one to properly handle memory effects in spike statistics, for large-sized neural networks. [ABSTRACT FROM AUTHOR]

Recently, the accuracy of voice authentication system has increased significantly due to the successful application of the identity vector (i-vector) model. This paper proposes a new method for i-vector extraction. In the method, a perceptual wavelet packet transform (PWPT) is designed to convert speech utterances into wavelet entropy feature vectors, and a Convolutional Neural Network (CNN) is designed to estimate the frame posteriors of the wavelet entropy feature vectors. In the end, i-vector is extracted based on those frame posteriors. TIMIT and VoxCeleb speech corpus are used for experiments and the experimental results show that the proposed method can extract appropriate i-vector which reduces the equal error rate (EER) and improve the accuracy of voice authentication system in clean and noisy environment. [ABSTRACT FROM AUTHOR]

The sustainability of regional water resources has important supporting data needed for establishing policies on the sustainable development of the social economy. The purpose of this paper is to propose an assessment method to accurately reflect the sustainability of regional water resources in various areas. The method is based on the relative entropy of the information entropy theory. The steps are as follows. Firstly, the pretreatment of the evaluation sample data is required, before the relative entropy of each standard evaluation sample and evaluation grade (SEG) is calculated to obtain the entropy weight of each evaluation index. After this, the entropy weighted comprehensive index (WCI) of the standard evaluation grade sample is obtained. The function relation between WCI and SEG can be fitted by the cubic polynomial to construct the evaluation function. Using the above steps, a generalized entropy method (GEM) for the sustainable assessment of regional water resources is established and it is used to evaluate the sustainability of water resources in the Pingba and Huai River areas in China. The results show that the proposed GEM model can accurately reflect the sustainable water resources in the two regions. Compared with the other evaluation models, such as the Shepherd method, Artificial Neural Network and Fuzzy comprehensive evaluation, the GEM model has larger differences in its evaluation results, which are more reasonable. Thus, the proposed GEM model can provide scientific data support for coordinating the relationship between the sustainable development and utilization of regional water resources in order to improve the development of regional population, society and economy. [ABSTRACT FROM AUTHOR]

Sensor fault diagnosis is necessary to ensure the normal operation of a gas turbine system. However, the existing methods require too many resources and this need can't be satisfied in some occasions. Since the sensor readings are directly affected by sensor state, sensor fault diagnosis can be performed by extracting features of the measured signals. This paper proposes a novel fault diagnosis method for sensors based on wavelet entropy. Based on the wavelet theory, wavelet decomposition is utilized to decompose the signal in different scales. Then the instantaneous wavelet energy entropy (IWEE) and instantaneous wavelet singular entropy (IWSE) are defined based on the previous wavelet entropy theory. Subsequently, a fault diagnosis method for gas turbine sensors is proposed based on the results of a numerically simulated example. Then, experiments on this method are carried out on a real micro gas turbine engine. In the experiment, four types of faults with different magnitudes are presented. The experimental results show that the proposed method for sensor fault diagnosis is efficient. [ABSTRACT FROM AUTHOR]

Understanding how ensembles of neurons collectively interact will be a key step in developing a mechanistic theory of cognitive processes. Recent progress in multineuron recording and analysis techniques has generated tremendous excitement over the physiology of living neural networks. One of the key developments driving this interest is a new class of models based on the principle of maximum entropy. Maximum entropy models have been reported to account for spatial correlation structure in ensembles of neurons recorded from several different types of data. Importantly, these models require only information about the firing rates of individual neurons and their pairwise correlations. If this approach is generally applicable, it would drastically simplify the problem of understanding how neural networks behave. Given the interest in this method, several groups now have worked to extend maximum entropy models to account for temporal correlations. Here, we review how maximum entropy models have been applied to neuronal ensemble data to account for spatial and temporal correlations. We also discuss criticisms of the maximum entropy approach that argue that it is not generally applicable to larger ensembles of neurons. We conclude that future maximum entropy models will need to address three issues: temporal correlations, higher-order correlations, and larger ensemble sizes. Finally, we provide a brief list of topics for future research. [ABSTRACT FROM AUTHOR]

Thresholding algorithms segment an image into two parts (foreground and background) by producing a binary version of our initial input. It is a complex procedure (due to the distinctive characteristics of each image) which often constitutes the initial step of other image processing or computer vision applications. Global techniques calculate a single threshold for the whole image while local techniques calculate a different threshold for each pixel based on specific attributes of its local area. In some of our previous work, we introduced some specific fuzzy inclusion and entropy measures which we efficiently managed to use on both global and local thresholding. The general method which we presented was an open and adaptable procedure, it was free of sensitivity or bias parameters and it involved image classification, mathematical functions, a fuzzy symmetrical triangular number and some criteria of choosing between two possible thresholds. Here, we continue this research and try to avoid all these by automatically connecting our measures with the wanted threshold using some Artificial Neural Network (ANN). Using an ANN in image segmentation is not uncommon especially in the domain of medical images. However, our proposition involves the use of an Adaptive Neuro-Fuzzy Inference System (ANFIS) which means that all we need is a proper database. It is a simple and immediate method which could provide researchers with an alternative approach to the thresholding problem considering that they probably have at their disposal some appropriate and specialized data. [ABSTRACT FROM AUTHOR]

The modeling and forecasting of BBS (Bulletin Board System) posts time series is crucial for government agencies, corporations and website operators to monitor public opinion. Accurate prediction of the number of BBS posts will assist government agencies or corporations in making timely decisions and estimating the future number of BBS posts will help website operators to allocate resources to deal with the possible hot events pressure. By combining sample entropy (SampEn) and deep neural networks (DNN), an approach (SampEn-DNN) is proposed for BBS posts time series modeling and forecasting. The main idea of SampEn-DNN is to utilize SampEn to decide the input vectors of DNN with smallest complexity, and DNN to enhance the prediction performance of time series. Selecting Tianya Zatan new posts as the data source, the performances of SampEn-DNN were compared with auto-regressive integrated moving average (ARIMA), seasonal ARIMA, polynomial regression, neural networks, etc. approaches for prediction of the daily number of new posts. From the experimental results, it can be found that the proposed approach SampEn-DNN outperforms the state-of-the-art approaches for BBS posts time series modeling and forecasting. [ABSTRACT FROM AUTHOR]