Search

The architecture of AND/OR fuzzy neural networks exhibits outstanding learning abilities and significant interpretation capabilities. However, AND/OR networks suffer from structure-related problems namely low efficiency and slow convergence of learning due to several reasons such as high dimensionality and gradient-based learning algorithms which lead to a visible computing overhead. In this paper, we present a two-phase fuzzy logic-oriented network design that is composed of AND/OR neurons. This design takes advantages of Randomized Neural Network (RNN) to achieve higher convergence while exhibiting good nonlinear approximation capabilities. A gradient–based learning algorithm is implemented in the second phase of the design to further reduce values of performance index. The quality of the proposed design and resulting architecture is quantified through the use of numeric data along with fuzzy sets (information granules). Experimental results meet the research's objectives and the proposed design methodology opens up new future directions for proceeding with more improvements. [ABSTRACT FROM AUTHOR]

In this study, a design methodology based on fuzzy relation-based neural networks for stabilized network structure is introduced to cope with over-fitting as well as multi-collinearity problems which generally appear in conventional fuzzy neural networks. The design method of the proposed self-organizing network structure provides an efficient solution to construct the stabilized Fuzzy Relation-based Neural Networks (FRNN) through a synergy of multi-techniques such as ensemble neurons/layers, L2-norm regularization, compromise technique, and multi-optimization, in order to generate the structure of the multi-layered self-organizing network designed with the aid of the learning as well as novel structural design. The overall network structure is realized with the aid of parallel network structure with newly added layers as well as effective node selection method through the combination technique of both a sum of squared coefficients (SSC) and performance index (PI) as a node selection criterion for each layer in FRNN. Ensemble neurons consist of the current inputs and the original inputs, and they serve to reduce the bias of the proposed FRNN by maintaining the information of the original inputs. Also, ensemble layers stand for the combination of the current layer and the front layer, and they play a role for control the variance of the model by taking into account the output of the previous layers. The least square error estimation (LSE)-based learning method with L2-norm regularization is used for constructing the stabilized network architecture, and their ensuring design methodologies result in alleviating the overfitting phenomenon and also enhancing the generalization ability. For the performance enhancement of FRNN directly affected by some parameters such as the number of input variables, collocation of the specific subset of input variables, the number of membership functions per each variable, and the order of polynomial in the consequent parts of the fuzzy rules, multi-particle swarm optimization (MPSO) is exploited for the effectively structural as well as parametric optimization of the proposed networks. That is, the multi-optimization helps achieve a compromise between the better generation performance and the alleviated over-fitting leading to the stabilization of the proposed multi-layered self-organizing network structure realized with the aid of synergistic multi-techniques and ensemble structures such as a) L2-norm regularization-based LSE learning, b) combination technique for effective node selection through the combination of SSC and PI, and c) a novel parallel network structure including newly added layers and node selection method. The performance of the proposed network structure is quantified by comprehensive experiments and comparative analysis. [ABSTRACT FROM AUTHOR]

The study is concerned with the problem of classification of multivariate time series using convolutional neural networks (CNNs). As CNNs regard inputs in the form of images, an original image-like format of temporal data is proposed. Along this line, several design alternatives are studied by forming images with the two corresponding coordinates built by the original temporal data and their differences and second differences. An overall design process is presented with a focus on investigating time series-image transformations. Experimental studies involving publicly available data sets are reported, along with a slew of comparative analyses. [ABSTRACT FROM AUTHOR]

• We propose fuzzy reinforced polynomial neural networks based on PNN architecture. • We propose fuzzy reinforced polynomial neurons (FRPNs) in the design of FRPNNs. • The proposed FRPNNs is a generalization of hybrid predictors (HPs). In the field of dynamic system identification and prediction, linear models (e.g., autoregressive models), nonlinear models (namely, neural networks models), and hybrid predictors (HPs) that are a hybridization of linear and nonlinear models have been proposed in the past. However, they are not completely free from limitations: they exhibit difficulties to describe high-order nonlinear relations between input and output variables. In this study, we propose fuzzy reinforced polynomial neural networks (FRPNNs), which are polynomial neural network architecture-based on fuzzy reinforced polynomial neurons (FRPNs) to overcome this limitation. The proposed FRPNs that consist of approximation part (AP) and compensation part (CP) arise as novel HPs. Here the CP for modeling nonlinear patterns can be regarded as forming the reinforced part for the AP that aims at capturing linear patterns, while AP is the linear polynomial neuron used in the conventional polynomial neural networks. In some sense, the overall FRPNNs are essentially generalized polynomial neural network architecture with novel HPs. The parameters considered in the design of the proposed fuzzy reinforced polynomial neural networks are optimized with the aid of the particle swarm optimization (PSO). The performance of FRPNNs is discussed involving time series and system identification datasets. Experimental results demonstrate that the proposed FRPNNs achieve at most the accuracy of 43.6% higher in comparison with the accuracy produced by some classical models reported in the literature. [ABSTRACT FROM AUTHOR]

Conventional artificial neural networks are inherently equipped with an ambiguous (uncertain) structure which is hard to be quantified and explained. Time series forecasting using neural networks thus becomes a highly challenging task also due to the fact that time series data are always nonlinear and uncertain (because of some disturbances). Considering this, we propose a granular neural network-based time series prediction model connecting the uncertainty of models and data with the concept of information granularity. We aim to provide an explainable time series prediction model to resist the disturbance inner time series data and reduce the vagueness of the model. The functionalities of the granular neural network model are threefold: (1) It reveals the uncertainty of a time series data set through the level of granularity , coverage and specificity and possesses high prediction accuracy; (2) It provides an optimized interval output endowed with enough specificity and sufficient coverage and this interval is more robust than a single value; (3) It owns an interpretable and flexible structure that reflects the uncertainty of the initial numeric neural network and the generalization and robustness of connections while being faced with disturbances. The experimental studies elaborate on each function of our model in detail and show that the developed method performed better than the existing approaches present in the literature when experimenting on several time series data sets. [ABSTRACT FROM AUTHOR]

Transformers have become the state-of-the-art architectures for various tasks in Natural Language Processing (NLP) and Computer Vision (CV); however, their space and computational complexity present significant challenges for real-world applications. A promising approach to address these issues is the introduction of sparsity, which involves the deliberate removal of certain parameters or activations from the neural network. In this systematic literature review, we aimed to provide a comprehensive overview of current research on sparsity in transformers. We analyzed the different sparsity techniques applied to transformers, their impact on model performance, and their efficiency in terms of time and space complexity. Moreover, we identified the major gaps and challenges in the existing literature. Our study also highlighted the importance of investigating sparsity in transformers for computational efficiency, reduced resource requirements, scalability, environmental impact, and hardware-algorithm co-design. By synthesizing the current state of research on sparsity in transformer-based models, we also provided valuable insights into their efficiency, impact on model performance, and potential trade-offs, contributing to advancing the field further. [ABSTRACT FROM AUTHOR]

In recent years, classical fuzzy clustering-based neural networks (FCNNs) have been successfully applied to regression tasks. The determination of the parameters such as cluster centers of the existing hard c-means (HCM) or fuzzy c-means (FCM), leads to the performance deterioration of the model because of the sensitivity of HCM or FCM to noise and outliers. Moreover, there are also several factors for over-fitting and degradation of the robustness of the ensuing model. To solve such problems, two improved clustering techniques and L 2 norm-regularization are considered in the proposed robust fuzzy clustering-based neural networks (RFCNNs) modeling. SVs-based hard c-means (SVs-based HCM) and SVs-based fuzzy c-means (SVs-based FCM) designed with support vectors (SVs) can reduce the interference of uncorrelated data, including noise and outliers, thereby enhancing the main data characteristics effectively, as well as leading to the construction on the improved network model. L 2 norm-regularization can be used to alleviate the degradation of robustness caused by overfitting. In terms of improving the performance of the model through SVs-based HCM or SVs-based FCM, as well as robustness completed through L 2 norm-regularization, the superiority of RFCNNs was verified by experimenting with synthetic data and publicly available data from machine learning datasets. [ABSTRACT FROM AUTHOR]

• A simple yet effective oversampling ensemble method for imbalanced data is proposed. • The sensitivity measure can compute impacts of class imbalance on training data. • Theoretical analyses for data generation are provided. • Proposed method is effective in handling datasets with various imbalance ratios. • The finding is ensembling perturbation-based oversamplers shows promising results. Oversampling is effective to handle imbalanced data and is often combined with ensemble learning to boost its performance, for examples the synthetic minority oversampling technique (SMOTE) and its ensemble variants SMOTEBagging and SMOTEBoost. However, SMOTE-based methods often introduce unnecessary noises when classes are not well separated, which may impinge upon classifier training and hinder the classification performance. In addition, the aggressive generation of SMOTE-based methods would further complicate the learning process. Therefore, we propose a simple yet effective approach to generate diverse instances to balance the class distribution in a more conservative way in this work. Major differences among the proposed method and existing methods, which also highlight the originality of this work, are listed as follows. Firstly, unlike existing methods which utilize nearest neighbors-based methods to determine regions of interest, we analyze and compute the impact of class imbalance quantitatively for each minority instance in order to guide the data generation process. This ensures instances can be synthesized in a safe and informative region to avoid noise generation. Secondly, new instances are generated via perturbing a random subsets of input features of given seed instances. We show that this process generates instances that share the same asymptotic mean and covariance as given seed instances, thus classifiers with high generalization capability can be achieved. The generation process is repeated to train multiple base classifiers which are then fused via a majority voting to further improve its performance. Nonparametric Wilcoxon test and Friedman test confirm that the proposed method significantly outperforms most reference methods over thirty-five imbalanced datasets in terms of five metrics. The important finding of this work is that we show that ensembling simple perturbation-based oversamplers can yield better performance than many advanced ensemble methods for imbalanced datasets. [ABSTRACT FROM AUTHOR]

This article develops a multi-view multi-label learning for incomplete data which are ubiquitous with the usage of three kinds of representations including within-view representation, cross-view representation, and consensus-view representation. Different from the recent learning machines, the proposed learning machine takes the feature-oriented information, label-oriented information, and associated information between features and labels in multiple representations together and exploits the hidden useful information of available instances with the usage of instance–instance correlations, feature–feature correlations, label-label correlations, and feature–label correlations. The developed learning machine is named as within- cross- consensus-view representation-based multi-view multi-label learning with incomplete data (WCC-MVML-ID). Extensive experiments on multiple multi-view and multi-label data sets with incomplete data validate the effectiveness of WCC-MVML-ID and it can be concluded that (1) WCC-MVML-ID outperforms other compared learning machines and its performances are more stable even though the missing rates of features and labels being larger; (2) compared with within-view information and consensus-view information, cross-view information is more useful for the processing problem about incomplete data; (3) WCC-MVML-ID can converge within 45 iterations. [ABSTRACT FROM AUTHOR]