Search

Recent years have witnessed the emergence of many new mobile apps and user-centered systems that interact with users by offering choices with rewards. These applications have been promising to address challenging societal problems such as congestion in transportation and behavior changes for healthier lifestyles. Considerable research efforts have been devoted to model the user behaviors in these new applications. However, as real-world user data is often prone to data corruptions, the success of these models hinges on a robust learning method. Building on the recently proposed Latent Decision Threshold model, this article shows that, among the existing robust learning frameworks, the L0-norm-based framework can outperform other state-of-the-art methods in terms of prediction accuracy and model estimation. And based on the L0-norm framework, we further develop a user screening algorithm to identify potential bad actors. [ABSTRACT FROM AUTHOR]

Huber regression plays a critical role in robust learning owing to its robustness to outliers and heavy-tailed noises. In this paper, we investigate the theoretical foundation of an Ivanov regularized Huber regression within a Reproducing Kernel Hilbert Space (RKHS) framework. By imposing a weak moment condition on the conditional distribution, we show that Huber regression estimator with the scale parameter adapted to sample size can learn the underlying regression function very well. Specifically, we establish the explicit convergence rates of the prediction error even when the regression function is outside the associated RKHS. Our statistical analysis could greatly improve the understanding of the learning behavior of Huber regression in the absence of light-tailed assumption on the conditional distribution. [ABSTRACT FROM AUTHOR]

Data containing noises, outliers, and imbalanced class distributions pose challenges to the traditional classifiers. By incorporating both the membership and non-membership functions, the intuitionistic fuzzy (IF) set has shown potential in designing robust learning algorithms for classifiers. However, the non-membership function used in these IF-based classifiers usually only utilizes the local distribution information of the training samples, and the classifiers are built upon single-hidden layer networks, which degrade the performance of the corresponding classifiers. Broad learning system (BLS) is an emerging neural network model with fast learning speed and flexible network architecture; however, it still fails to distinguish n samples. To this end, in this paper, we propose a new definition of the non-membership function within intuitionistic fuzzy sets and subsequently propose an intuitionistic fuzzy broad learning system (IFBLS) model. The proposed non-membership function incorporates two ratio numbers based on four distances, allowing for the utilization of global information on the distribution of samples and mitigating misclassification of valid samples as noise which is often observed in traditional methods. By using a score function that considers both the membership and non-membership functions to redistribute the importance of the training samples, the proposed IFBLS benefits from both the powerful representation capability of the original BLS and the robust learning of IF-based models. Extensive experiments conducted on 21 imbalanced binary classification problems sourced from the UCI and KEEL repositories illustrate that the proposed IFBLS achieves state-of-the-art performance by attaining the highest testing accuracy in 17 out of the 21 problems. [ABSTRACT FROM AUTHOR]

Numerous Deep Learning (DL) scenarios have been developed for evolving new healthcare systems that leverage large datasets, distributed computing, and the Internet of Things (IoT). However, the data used in these scenarios tend to be noisy, necessitating the incorporation of robust pre-processing techniques, including data cleaning, preparation, normalization, and addressing imbalances. These steps are crucial for generating a robust dataset for training. Designing frameworks capable of handling such data without compromising efficiency is essential to ensuring robustness. This research aims to propose a novel healthcare framework that selects the best features and enhances performance. This robust deep learning framework, called (R-DLH2O), is designed for forecasting cardiovascular crises. Unlike existing methods, R-DLH2O integrates five distinct phases: robust pre-processing, feature selection, feed-forward neural network, prediction, and performance evaluation. This multi-phase approach ensures superior accuracy and efficiency in crisis prediction, offering a significant advancement in healthcare analytics. H2O is utilized in the R-DLH2O framework for processing big data. The main improvement of this paper lies in the unique form of the Whale Optimization Algorithm (WOA), specifically the Modified WOA (MWOA). The Gaussian distribution approach for random walks was employed with the diffusion strategy to choose the optimal MWOA solution during the growth phase. To validate the R-DLH2O framework, six performance tests were conducted. Surprisingly, the MWOA-2 outperformed other heuristic algorithms in speed, despite exhibiting lower accuracy and scalability. The suggested MWOA was further analyzed using benchmark functions from CEC2005, demonstrating its advantages in accuracy and robustness over WOA. These findings highlight that the framework's processing time is 436 s, mean per-class error is 0.150125, accuracy 95.93%, precision 92.57%, and recall 93.6% across all datasets. These findings highlight the framework's potential to produce significant and robust results, outperforming previous frameworks concerning time and accuracy. [ABSTRACT FROM AUTHOR]

The development and application of artificial intelligence-based computer vision systems in medicine, environment, and industry are playing an increasingly prominent role. Hence, the need for optimal and efficient hyperparameter tuning strategies is more than crucial to deliver the highest performance of the deep learning networks in large and demanding datasets. In our study, we have developed and evaluated a new training methodology named deep multi-metric training (DMMT) for enhanced training performance. The DMMT delivers a state of robust learning for deep networks using a new important criterion of multi-metric performance evaluation. We have tested the DMMT methodology in multi-class (three, four, and ten), multi-vendors (different X-ray imaging devices), and multi-size (large, medium, and small) datasets. The validity of the DMMT methodology has been tested in three different classification problems: (i) medical disease classification, (ii) environmental classification, and (iii) ecological classification. For disease classification, we have used two large COVID-19 chest X-rays datasets, namely the BIMCV COVID-19+ and Sheffield hospital datasets. The environmental application is related to the classification of weather images in cloudy, rainy, shine or sunrise conditions. The ecological classification task involves a classification of three animal species (cat, dog, wild) and a classification of ten animals and transportation vehicles categories (CIFAR-10). We have used state-of-the-art networks of DenseNet-121, ResNet-50, VGG-16, VGG-19, and DenResCov-19 (DenRes-131) to verify that our novel methodology is applicable in a variety of different deep learning networks. To the best of our knowledge, this is the first work that proposes a training methodology to deliver robust learning, over a variety of deep learning networks and multi-field classification problems. [ABSTRACT FROM AUTHOR]

Tremendous breakthroughs have been developed in Semi-Supervised Semantic Segmentation (S4) through contrastive learning. However, due to limited annotations, the guidance on unlabeled images is generated by the model itself, which inevitably exists noise and disturbs the unsupervised training process. To address this issue, we propose a robust contrastive-based S4 framework, termed the Probabilistic Representation Contrastive Learning (PRCL) framework to enhance the robustness of the unsupervised training process. We model the pixel-wise representation as Probabilistic Representations (PR) via multivariate Gaussian distribution and tune the contribution of the ambiguous representations to tolerate the risk of inaccurate guidance in contrastive learning. Furthermore, we introduce Global Distribution Prototypes (GDP) by gathering all PRs throughout the whole training process. Since the GDP contains the information of all representations with the same class, it is robust from the instant noise in representations and bears the intra-class variance of representations. In addition, we generate Virtual Negatives (VNs) based on GDP to involve the contrastive learning process. Extensive experiments on two public benchmarks demonstrate the superiority of our PRCL framework. [ABSTRACT FROM AUTHOR]

Abstract Numerous Deep Learning (DL) scenarios have been developed for evolving new healthcare systems that leverage large datasets, distributed computing, and the Internet of Things (IoT). However, the data used in these scenarios tend to be noisy, necessitating the incorporation of robust pre-processing techniques, including data cleaning, preparation, normalization, and addressing imbalances. These steps are crucial for generating a robust dataset for training. Designing frameworks capable of handling such data without compromising efficiency is essential to ensuring robustness. This research aims to propose a novel healthcare framework that selects the best features and enhances performance. This robust deep learning framework, called (R-DLH2O), is designed for forecasting cardiovascular crises. Unlike existing methods, R-DLH2O integrates five distinct phases: robust pre-processing, feature selection, feed-forward neural network, prediction, and performance evaluation. This multi-phase approach ensures superior accuracy and efficiency in crisis prediction, offering a significant advancement in healthcare analytics. H2O is utilized in the R-DLH2O framework for processing big data. The main improvement of this paper lies in the unique form of the Whale Optimization Algorithm (WOA), specifically the Modified WOA (MWOA). The Gaussian distribution approach for random walks was employed with the diffusion strategy to choose the optimal MWOA solution during the growth phase. To validate the R-DLH2O framework, six performance tests were conducted. Surprisingly, the MWOA-2 outperformed other heuristic algorithms in speed, despite exhibiting lower accuracy and scalability. The suggested MWOA was further analyzed using benchmark functions from CEC2005, demonstrating its advantages in accuracy and robustness over WOA. These findings highlight that the framework’s processing time is 436 s, mean per-class error is 0.150125, accuracy 95.93%, precision 92.57%, and recall 93.6% across all datasets. These findings highlight the framework’s potential to produce significant and robust results, outperforming previous frameworks concerning time and accuracy.

In this study, a corrective learning technique is proposed to enhance the time series forecasting accuracy of traditional intelligent algorithms and address issues with engineering data adaptability. The financial world makes substantial use of volatility, and forecasting stock market volatility has great significance. Because stock price time series are nonstationary and nonlinear, predicting stock market volatility is a significant difficulty. In this study, a simulation experiment using an approach for corrective learning is carried out. Over a 3-year period, a LightGBM-GRU model is created to predict the volatility of 126 equities from diverse industries over the length of the subsequent 10 min, utilizing LightGBM and GRU as base models and correctors, respectively. Even traditional and effective ensemble learning models can benefit from the corrective learning technique, which simultaneously improves accuracy and generalization, according to performance metrics like RMSPE, MAE, MSE, and RMSE. The research emphasizes how the contradiction within intelligent algorithms has changed from one of restricted generality to different engineering difficulties in the era of algorithm enrichment and large data. The corrective learning strategy offers a fresh method for simulating data. [ABSTRACT FROM AUTHOR]

Due to the exceptional learning capabilities of deep neural networks (DNNs), they continue to struggle to handle label noise. To address this challenge, the pseudo-label approach has emerged as a preferred solution. Recent works have achieved significant improvements by exploring the information involved in DNN predictions and designing a straightforward method to incorporate model predictions into the training process by using a convex combination of original labels and predictions as the training targets. However, these methods overlook the feature-level information contained in the sample, which significantly impacts the accuracy of the pseudo label. This study introduces a straightforward yet potent technique named FPL (feature pseudo-label), which leverages information from model predictions as well as feature similarity. Additionally, we utilize an exponential moving average scheme to bolster the stability of corrected labels while upholding the stability of pseudo-labels. Extensive experiments were carried out on synthetic and real datasets across different noise types. The CIFAR10 dataset yielded the highest accuracy of 94.13% (Top1), while Clothing1M achieved 73.54%. The impressive outcomes showcased the efficacy and robustness of learning amid label noise. [ABSTRACT FROM AUTHOR]

Recent advancements in offline reinforcement learning (RL) have showcased the potential for leveraging static datasets to train optimal policies. However, real-world applications often face challenges due to missing or incomplete state information caused by imperfect sensor performance or intentional interlaces. We propose the Dimension-Wise Drop Decision Transformer (D3T), a novel framework designed to address dimension-wise data loss in sensor observations, enhancing the robustness of RL algorithms in real-world scenarios. D3T innovatively incorporates dimension-wise drop information embeddings within the Transformer architecture, facilitating effective decision-making even with incomplete observations. Our evaluation in the D4RL MuJoCo domain demonstrates that D3T significantly outperforms existing methods such as the Decision Transformer, particularly with substantial dimension-wise drops of observations. These results confirm D3T’s capability in managing real-world imperfections in state observations and illustrate its potential to substantially expand the applicability of RL in more complex and dynamic environments.

Physics-Informed Neural Network (PINN) has emerged as a promising tool for solving various physical problems with differential equations. However in practice, PINN often suffers from the local minima issue while solving problems with minimal initial conditions. In this paper, we propose a robust Gated-PINN to address this issue, which blends numerical methods and PINN. The Gated-PINN overcomes the local minima issue by controlling the flow of information of the neural network similar to a numerical method. We first investigate the use of conventional PINN to the Cartesian-coordinate simple pendulum problem with minimal initial conditions, as one of basic differential algebraic equation (DAE) problem. Our results show the advantage of PINN over existing numerical methods in that it does not require sophisticated mathematical techniques such as index reduction. But we also observe that conventional PINN can lead to inaccurate solutions; such solutions partially satisfy the differential equation requirement, but does not meet the given initial condition and fail to further improve. We demonstrate the effectiveness of the proposed Gated-PINN by showing that it yields accurate solutions, such that for a pendulum of length 1, the mean Euclidean error between the Gated-PINN model and the traditional numerical method model is less than 0.01 and pointwise maximum Euclidean error is less than 0.04. Moreover, Gated-PINN can operate without any complicated index reduction, and unlike conventional PINN, accurate solution can be obtained consistently without falling into a local minima. Overall, our study presents the potential of the Gated-PINN for solving DAE problems and provides a valuable insight into the challenges and limitations of using PINN for solving physical problems.

Copyright of Frontiers of Information Technology & Electronic Engineering is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In many online platforms, customers' decisions are substantially influenced by product rankings asmost customers only examine a fewtop-ranked products. This induces a race for visibility among sellers, who may be incentivized to artificially inflate their position by employing fake users as exemplified by the emergence of click farms. Motivated by such fraudulent behavior, we study the problem of learning product rankings when a platform faces a mixture of real and fake users who are indistinguishable from one another. We first show that existing learning algorithms--that are optimal in the absence of fake users--may converge to highly suboptimal rankings under manipulation. To overcome this deficiency, we develop efficient learning algorithms under two informational settings: when the platform is aware of the number of fake users and when it is agnostic to this number. For both these settings, we prove that our algorithms converge to the optimal ranking, yet being robust to the aforementioned fraudulent behavior; we also present worst case performance guarantees for our methods and show that they outperform existing algorithms. At a high level, our work employs several novel approaches to guarantee robustness, such as (i) encoding product relationships using graphs and (ii) implementing multiple levels of learning as well as judicious cross-learning. Overall, our results indicate that online platforms can effectively combat fraudulent users even when they are completely oblivious to the number and identity of the fake users. [ABSTRACT FROM AUTHOR]

Due to the exceptional learning capabilities of deep neural networks (DNNs), they continue to struggle to handle label noise. To address this challenge, the pseudo-label approach has emerged as a preferred solution. Recent works have achieved significant improvements by exploring the information involved in DNN predictions and designing a straightforward method to incorporate model predictions into the training process by using a convex combination of original labels and predictions as the training targets. However, these methods overlook the feature-level information contained in the sample, which significantly impacts the accuracy of the pseudo label. This study introduces a straightforward yet potent technique named FPL (feature pseudo-label), which leverages information from model predictions as well as feature similarity. Additionally, we utilize an exponential moving average scheme to bolster the stability of corrected labels while upholding the stability of pseudo-labels. Extensive experiments were carried out on synthetic and real datasets across different noise types. The CIFAR10 dataset yielded the highest accuracy of 94.13% (Top1), while Clothing1M achieved 73.54%. The impressive outcomes showcased the efficacy and robustness of learning amid label noise.

Semi-supervised learning (SSL) methods provide a powerful tool for utilizing abundant unlabeled data to strengthen standard supervised learning. Traditional graph-based SSL methods prevail in classical SSL problems for their intuitional implementation and effective performance. However, they encounter troubles when applying to image classification followed by modern deep learning, since the diffusion algorithms face the curse of dimensionality. In this study, we propose a simple and efficient SSL method, combining a graph-based SSL paradigm with differential privacy. We aim at developing coherent latent feature space of deep neural networks so that the diffusion algorithm in the latent space can give more precise predictions for unlabeled data. Our approach achieves state-of-the-art performance on the Cifar10, Cifar100, and Mini-imagenet benchmark datasets and obtains an error rate of 18.56% on Cifar10 using only 1% of all labels. Furthermore, our approach inherits the benefits of graph-based SSL methods with a simple training process and can be easily combined with any network architecture.

The use of multi-agent systems to solve large-scale problems can be an effective method to reduce physical and computational burdens; however, these systems should be robust to sub-system failures. In this work, we consider the problem of designing utility functions, which agents seek to maximize, as a method of distributed optimization in resource allocation problems. Though recent work has shown that optimal utility design can bring system operation into a reasonable approximation of optimal, our results extend the existing literature by investigating how robust the system's operation is to defective agents and by quantifying the achievable performance guarantees in this setting. Interestingly, we find that there is a trade-off between improving the robustness of the utility design and offering good nominal performance. We characterize this trade-off in the set of resource covering problems and find that there are considerable gains in robustness that can be made by sacrificing some nominal performance. [ABSTRACT FROM AUTHOR]

Outliers in the training or test set used to fit and evaluate a classifier on transcriptomics data can considerably change the estimated performance of the model. Hence, an either too weak or a too optimistic accuracy is then reported and the estimated model performance cannot be reproduced on independent data. It is then also doubtful whether a classifier qualifies for clinical usage. We estimate classifier performances in simulated gene expression data with artificial outliers and in two real-world datasets. As a new approach, we use two outlier detection methods within a bootstrap procedure to estimate the outlier probability for each sample and evaluate classifiers before and after outlier removal by means of cross-validation. We found that the removal of outliers changed the classification performance notably. For the most part, removing outliers improved the classification results. Taking into account the fact that there are various, sometimes unclear reasons for a sample to be an outlier, we strongly advocate to always report the performance of a transcriptomics classifier with and without outliers in training and test data. This provides a more diverse picture of a classifier's performance and prevents reporting models that later turn out to be not applicable for clinical diagnoses. [ABSTRACT FROM AUTHOR]

The performance of deep neural networks (DNNs) critically relies on high-quality annotations, while training DNNs with noisy labels remains challenging owing to their incredible capacity to memorize the entire training set. In this work, we use two synchronously trained networks to reveal that noisy labels may result in more divergent gradients when updating the parameters. To overcome this, we propose a novel co-training framework named gradient agreement learning (GAL). By dynamically evaluating the gradient agreement coefficient of every pair of parameters from two identical DNNs to determine whether to update them in the training process. GAL can effectively hinder the memorization of noisy labels. Furthermore, we utilize the pseudo labels produced by the two DNNs as the supervision for the training of another network, thereby gaining further improvement by correcting some noisy labels while overcoming the confirmation bias. Extensive experiments on various benchmark datasets demonstrate the superiority of the proposed GAL. [ABSTRACT FROM AUTHOR]

Optimization problems with continuous data appear in, e.g., robust machine learning, functional data analysis, and variational inference. Here, the target function is given as an integral over a family of (continuously) indexed target functions--integrated with respect to a probability measure. Such problems can often be solved by stochastic optimization methods: performing optimization steps with respect to the indexed target function with randomly switched indices. In this work, we study a continuous-time variant of the stochas tic gradient descent algorithm for optimization problems with continuous data. This so-called stochastic gradient process consists in a gradient flow minimizing an indexed target function that is coupled with a continuous-time index process determining the index. Index processes are, e.g., reflected diffusions, pure jump processes, or other Lévy processes on compact spaces. Thus, we study multiple sampling patterns for the continuous data space and allow for data simulated or streamed at runtime of the algorithm. We analyze the approximation properties of the stochastic gradient process and study its longtime behavior and ergodicity under constant and decreasing learning rates. We end with illustrating the applicability of the stochastic gradient process in a polynomial regression problem with noisy functional data, as well as in a physics-informed neural network. [ABSTRACT FROM AUTHOR]

Robust learning is an important issue in Scientific Machine Learning (SciML). There are several works in the literature addressing this topic. However, there is an increasing demand for methods that can simultaneously consider all the different uncertainty components involved in SciML model identification. Hence, this work proposes a comprehensive methodology for uncertainty evaluation of the SciML that also considers several possible sources of uncertainties involved in the identification process. The uncertainties considered in the proposed method are the absence of a theory, causal models, sensitivity to data corruption or imperfection, and computational effort. Therefore, it is possible to provide an overall strategy for uncertainty-aware models in the SciML field. The methodology is validated through a case study developing a soft sensor for a polymerization reactor. The first step is to build the nonlinear model parameter probability distribution (PDF) by Bayesian inference. The second step is to obtain the machine learning model uncertainty by Monte Carlo simulations. In the first step, a PDF with 30,000 samples is built. In the second step, the uncertainty of the machine learning model is evaluated by sampling 10,000 values through Monte Carlo simulation. The results demonstrate that the identified soft sensors are robust to uncertainties, corroborating the consistency of the proposed approach. [ABSTRACT FROM AUTHOR]

In this work, we propose a framework in which we use a Lipschitz-constrained loss minimization scheme to learn feedback control policies with guarantees on closed-loop stability, adversarial robustness, and generalization. These policies are learned directly from expert demonstrations, contained in a dataset of state-control input pairs, without any prior knowledge of the task and system model. Our analysis exploits the Lipschitz property of the learned policies to obtain closed-loop guarantees on stability, adversarial robustness, and generalization over scenarios unexplored by the expert. In particular, first, we establish robust closed-loop stability under the learned control policy, where we provide guarantees that the closed-loop trajectory under the learned policy stays within a bounded region around the expert trajectory and converges asymptotically to a bounded region around the origin. Second, we derive bounds on the closed-loop regret with respect to the expert policy and on the deterioration of the closed-loop performance under bounded (adversarial) disturbances to the state measurements. These bounds provide certificates for closed-loop performance and adversarial robustness for learned policies. Third, we derive a (probabilistic) bound on generalization error for the learned policies. Numerical results validate our analysis and demonstrate the effectiveness of our robust feedback policy learning framework. Finally, our results support the existence of a potential tradeoff between nominal closed-loop performance and adversarial robustness, and that improvements in nominal closed-loop performance can only be made at the expense of robustness to adversarial perturbations.

Deep neural networks (DNNs) can achieve high accuracy when there is abundant training data that has the same distribution as the test data. In practical applications, data deficiency is often a concern. For classification tasks, the lack of enough labeled images in the training set often results in overfitting. Another issue is the mismatch between the training and the test domains, which results in poor model performance. This calls for the need to have robust and data efficient deep learning models. In this work, we propose a deep learning approach called Multi-Expert Adversarial Regularization learning (MEAR) with limited computational overhead to improve the generalization and robustness of deep supervised learning models. The MEAR framework appends multiple classifier heads (experts) to the feature extractor of the legacy model. MEAR aims to learn the feature extractor in an adversarial fashion by leveraging complementary information from the individual experts as well as the ensemble of the experts to be more robust for an unseen test domain. We train state-of-the-art networks with MEAR for two important computer vision tasks, image classification and semantic segmentation. We compare MEAR to a variety of baselines on multiple benchmarks. We show that MEAR is competitive with other methods and more successful at learning robust features.

Beside the minimizationof the prediction error, two of the most desirable properties of a regression scheme are stability and interpretability. Driven by these principles, we propose continuous-domain formulations for one-dimensional regression problems. In our first approach, we use the Lipschitz constant as a regularizer, which results in an implicit tuning of the overall robustness of the learned mapping. In our second approach, we control the Lipschitz constant explicitly using a user-defined upper-bound and make use of a sparsity-promoting regularizer to favor simpler (and, hence, more interpretable) solutions. The theoretical study of the latter formulation is motivated in part by its equivalence, which we prove, with the training of a Lipschitz-constrained two-layer univariate neural network with rectified linear unit (ReLU) activations and weight decay. By proving representer theorems, we show that both problems admit global minimizers that are continuous and piecewise-linear (CPWL) functions. Moreover, we propose efficient algorithms that find the sparsest solution of each problem: the CPWL mapping with the least number of linear regions. Finally, we illustrate numerically the outcome of our formulations.

The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning. [ABSTRACT FROM AUTHOR]

As a novel similarity measure that is defined as the expectation of a kernel function between two random variables, correntropy has been successfully applied in robust machine learning and signal processing to combat large outliers. The kernel function in correntropy is usually a zero-mean Gaussian kernel. In a recent work, the concept of mixture correntropy (MC) was proposed to improve the learning performance, where the kernel function is a mixture Gaussian kernel, namely, a linear combination of several zero-mean Gaussian kernels with different widths. In both correntropy and MC, the center of the kernel function is, however, always located at zero. In the present work, to further improve the learning performance, we propose the concept of multikernel correntropy (MKC), in which each component of the mixture Gaussian kernel can be centered at a different location. The properties of the MKC are investigated and an efficient approach is proposed to determine the free parameters in MKC. Experimental results show that the learning algorithms under the maximum MKC criterion (MMKCC) can outperform those under the original maximum correntropy criterion (MCC) and the maximum MC criterion (MMCC). [ABSTRACT FROM AUTHOR]

A small portion of mislabeled data can easily limit the performance of deep neural networks (DNNs) due to their high capacity for memorizing random labels. Thus, robust learning from noisy labels has become a key challenge for deep learning due to inadequate datasets with high-quality annotations. Most existing methods involve training models on clean sets by dividing clean samples from noisy ones, resulting in large amounts of mislabeled data being unused. To address this problem, we propose categorizing training samples into five fine-grained clusters based on the difficulty experienced by DNN models when learning them and label correctness. A novel fine-grained confidence modeling (FGCM) framework is proposed to cluster samples into these five categories; with each cluster, FGCM decides whether to accept the cluster data as they are, accept them with label correction, or accept them as unlabeled data. By applying different strategies to the fine-grained clusters, FGCM can better exploit training data than previous methods. Extensive experiments on widely used benchmarks CIFAR-10, CIFAR-100, clothing1M, and WebVision with different ratios and types of label noise demonstrate the superiority of our FGCM. [ABSTRACT FROM AUTHOR]

Deep neural networks provide end-to-end tools to learn effective representations from data directly. The deep structure makes it possible to model a complicated pattern, even if it has a variety of changes. This leads to a problem that noises and outliers are usually treated as a specific pattern, which is also learned in the network. It is one reason for overfitting incapable of being addressed sufficiently in deep networks. This paper proposes a new method called subdomain contraction (SDC) to tackle the problem. The idea is that our approach inclines to learn more about the shared features between the subsets of the samples but less about the specific features found in only one or two subsets. To this end, the SDC loss penalizes the distribution distance between sub-domains in the feature space to constrain the sharing level of features. By applying the SDC loss term, the data drive the learning process to an optimal tradeoff between modeling noises and the varieties of the pattern. In this manner, the SDC models the pattern as much as possible and ignores most noises, thus improving the generalization ability. The SDC loss can be efficiently computed in minibatches and can also work collaboratively with other regularization methods such as dropout to further improve the performance. Extensive experiments demonstrate that SDC can improve the effectiveness and robustness of representation learning in deep networks against noises, and the superiority is most remarkable with noisy data. [ABSTRACT FROM AUTHOR]

In this paper, we study the performance of robust learning with Huber loss. As an alternative to traditional empirical risk minimization schemes, Huber regression has been extensively used in machine learning. A new comparison theorem is established in the paper, which characterizes the gap between the excess generalization error and the prediction error. In addition, we refine the error bounds from the perspective of statistical learning theory and improve the convergence rates in the presence of heavy-tailed noise. It is worth mentioning that a new moment condition E [ | Y | 1 + ∊ | X = x ] ∈ L ρ X 2 is employed in analysis of error bound and learning rates from a theoretical viewpoint. [ABSTRACT FROM AUTHOR]

We consider a distributionally robust formulation of stochastic optimization problems arising in statistical learning, where robustness is with respect to ambiguity in the underlying data distribution. Our formulation builds on risk-averse optimization techniques and the theory of coherent risk measures. It uses mean–semideviation risk for quantifying uncertainty, allowing us to compute solutions that are robust against perturbations in the population data distribution. We consider a broad class of generalized differentiable loss functions that can be non-convex and non-smooth, involving upward and downward cusps, and we develop an efficient stochastic subgradient method for distributionally robust problems with such functions. We prove that it converges to a point satisfying the optimality conditions. To our knowledge, this is the first method with rigorous convergence guarantees in the context of generalized differentiable non-convex and non-smooth distributionally robust stochastic optimization. Our method allows for the control of the desired level of robustness with little extra computational cost compared to population risk minimization with stochastic gradient methods. We also illustrate the performance of our algorithm on real datasets arising in convex and non-convex supervised learning problems. [ABSTRACT FROM AUTHOR]

The control-based approach has been proved to be effective for developing robust online learning methods. However, the existing control-based kernel methods are infeasible for large-scale modeling due to their high computational complexity. This paper aims to propose a computationally efficient control-based framework for robust large-scale kernel learning problems. By random feature approximation and robust loss function, the learning problems are first transformed into a group of linear feedback control problems with sparse discrete large-scale algebraic Riccati equations (DARE). Then, with the solutions of the DAREs, two promising algorithms are developed to address large-scale binary classification and regression problems, respectively. Thanks to the sparseness, explicit solutions rather than numerical solutions of the DAREs are derived by utilizing matrix computation techniques developed in our study. This substantially reduces the complexity, and makes the proposed algorithms computationally efficient for large-scale complex datasets. Compared with the existing benchmarks, the proposed algorithms can achieve faster convergent, more robust and accurate modeling results. Theoretical analysis and encouraging numerical results on synthetic and realistic datasets are also provided to illustrate the effectiveness and efficiency of our algorithms. [ABSTRACT FROM AUTHOR]

The memorization effect of deep neural networks (DNNs) plays a pivotal role in recent label noise learning methods. To exploit this effect, the model prediction-based methods have been widely adopted, which aim to exploit the outputs of DNNs in the early stage of learning to correct noisy labels. However, we observe that the model will make mistakes during label prediction, resulting in unsatisfactory performance. By contrast, the produced features in the early stage of learning show better robustness. Inspired by this observation, in this paper, we propose a novel feature embedding-based method for deep learning with label noise, termed LabElNoiseDilution (LEND). To be specific, we first compute a similarity matrix based on current embedded features to capture the local structure of training data. Then, the noisy supervision signals carried by mislabeled data are overwhelmed by nearby correctly labeled ones (i.e., label noise dilution), of which the effectiveness is guaranteed by the inherent robustness of feature embedding. Finally, the training data with diluted labels are further used to train a robust classifier. Empirically, we conduct extensive experiments on both synthetic and real-world noisy datasets by comparing our LEND with several representative robust learning approaches. The results verify the effectiveness of our LEND. [ABSTRACT FROM AUTHOR]

With the growth in complexity of real-time embedded systems, there is an increasing need for tools and techniques to understand and compare the observed runtime behavior of a system with the expected one. Since many real-time applications require periodic interactions with the environment, one of the fundamental problems in guaranteeing their temporal correctness is to be able to infer the periodicity of certain events in the system. The practicability of a period inference tool, however, depends on both its accuracy and robustness (also its resilience) against noise in the output trace of the system, e.g., when the system trace is impacted by the presence of aperiodic tasks, release jitters, and runtime variations in the execution time of the tasks. This work (i) presents the first period inference framework that uses regression-based machine-learning (RBML) methods, and (ii) thoroughly investigates the accuracy and robustness of different families of RBML methods in the presence of uncertainties in the system parameters. We show, on both synthetically generated traces and traces from actual systems, that our solutions can reduce the error of period estimation by two to three orders of magnitudes w.r.t. the state of the art. [ABSTRACT FROM AUTHOR]

Label noises, categorized into closed-set noise and open-set noise, are prevalent in real-world scenarios and can seriously hinder the generalization ability of models. Identifying noise is challenging because noisy samples closely resemble true positives. Existing approaches often assume a single noise source, oversimplify closed-set noise, or treat open-set noise as toxic and eliminate it, resulting in limited practical effects. To address these issues, we present a novel approach named u ncertainty-guided l abel correction with wavelet- tra nsformed discriminative representation enhancement (Ultra), designed to mitigate the effects of mixed noise. Specifically, our approach considers a more practical noise setting. To achieve robust mixed-noise identification, we initially look into a learnable wavelet filter for obtaining discriminative features and filtering spurious cues automatically at the representation level. Subsequently, we introduce a two-fold uncertainty estimation to stably locate noise within the corrupted supervised signal level. These insights pave the way for a simple yet potent label correction technique, enabling comprehensive utilization of open-set noise, which can be rendered non-toxic in a specific manner, in contrast to harmful closed-set noise. Experimental validation on datasets with synthetic mixed noise, web noise corruption, and a real-world dataset confirms the effectiveness and generality of Ultra. Furthermore, our approach enhances the application of efficient techniques (e.g., supervised contrastive learning) within label noise scenarios. [ABSTRACT FROM AUTHOR]

Minimum error entropy (MEE) is an information theoretic learning approach that minimizes the information contained in the prediction error, which is measured by entropy. It has been successfully used in various machine learning tasks for its robustness to heavy-tailed distributions and outliers. In this paper, we consider its use in nonparametric regression and analyze its generalization performance from a learning theory perspective by imposing a (1 +) th order moment condition on the noise variable. To this end, we establish a comparison theorem to characterize the relation between the excess generalization error and the prediction error. A relaxed Bernstein condition and concentration inequalities are used to derive error bounds and learning rates. Note that the (1 +) th moment condition is rather weak particularly when < 1 because the noise variable does not even admit a finite variance in this case. Therefore, our analysis explains the robustness of MEE in the presence of heavy-tailed distributions. [ABSTRACT FROM AUTHOR]

Recently, extreme learning machine (ELM) has attracted a lot of attention due to its high performance and extreme speed. However, how to improve the robustness of ELM has always been a problem. Considering the problems of noise and outliers in the experimental data, in this paper, we introduce the kernel risk-sensitive mean p-power error (KRP) into ELM and propose a robust ELM method named kernel risk-sensitive mean p-power error based robust extreme learning machine (KRPELM), on the basis of the high efficiency and robustness of KRP. In KRPELM, KRP function instead of square loss is integrated into ELM as the loss function, which can improve the robustness of ELM to noise and outliers. We also propose an efficient iterative adjustment strategy to optimize KRPELM. Nine benchmark datasets are utilized to verify the classification performance of the proposed KRPELM. In addition, we apply the proposed method to the classification of cancer samples. The experimental results on five cancer gene expression datasets show that KRPELM can identify different cancer types more accurately. [ABSTRACT FROM AUTHOR]

In distributed wireless sensor networks, geographically distributed sensors cooperate wirelessly with each other. While sensing from the environment, the signals from these sensors are often contaminated by noise. Traditional diffusion algorithms for distributed estimation consider this noise to be Gaussian in nature. However, in practice this noise can also be non-Gaussian, which leads to deterioration in performance of traditional adaptive algorithms. Moreover, the parameter vector to be estimated may be sparse in nature. To improve adaptive filter performance for distributed networks, we propose a set of sparsity aware diffusion adaptive filters which are robust to non-Gaussian noises. Extensive simulation study for different Gaussian and non-Gaussian noise environments show the improved estimation ability of the proposed algorithms for modelling highly, moderate and non-sparse distributed systems. [ABSTRACT FROM AUTHOR]

The performance of deep neural networks (DNNs) critically relies on high-quality annotations, while training DNNs with noisy labels remains challenging owing to their incredible capacity to memorize the entire training set. In this work, we use two synchronously trained networks to reveal that noisy labels may result in more divergent gradients when updating the parameters. To overcome this, we propose a novel co-training framework named gradient agreement learning (GAL). By dynamically evaluating the gradient agreement coefficient of every pair of parameters from two identical DNNs to determine whether to update them in the training process. GAL can effectively hinder the memorization of noisy labels. Furthermore, we utilize the pseudo labels produced by the two DNNs as the supervision for the training of another network, thereby gaining further improvement by correcting some noisy labels while overcoming the confirmation bias. Extensive experiments on various benchmark datasets demonstrate the superiority of the proposed GAL.

Robust learning is an important issue in Scientific Machine Learning (SciML). There are several works in the literature addressing this topic. However, there is an increasing demand for methods that can simultaneously consider all the different uncertainty components involved in SciML model identification. Hence, this work proposes a comprehensive methodology for uncertainty evaluation of the SciML that also considers several possible sources of uncertainties involved in the identification process. The uncertainties considered in the proposed method are the absence of a theory, causal models, sensitivity to data corruption or imperfection, and computational effort. Therefore, it is possible to provide an overall strategy for uncertainty-aware models in the SciML field. The methodology is validated through a case study developing a soft sensor for a polymerization reactor. The first step is to build the nonlinear model parameter probability distribution (PDF) by Bayesian inference. The second step is to obtain the machine learning model uncertainty by Monte Carlo simulations. In the first step, a PDF with 30,000 samples is built. In the second step, the uncertainty of the machine learning model is evaluated by sampling 10,000 values through Monte Carlo simulation. The results demonstrate that the identified soft sensors are robust to uncertainties, corroborating the consistency of the proposed approach.

A small portion of mislabeled data can easily limit the performance of deep neural networks (DNNs) due to their high capacity for memorizing random labels. Thus, robust learning from noisy labels has become a key challenge for deep learning due to inadequate datasets with high-quality annotations. Most existing methods involve training models on clean sets by dividing clean samples from noisy ones, resulting in large amounts of mislabeled data being unused. To address this problem, we propose categorizing training samples into five fine-grained clusters based on the difficulty experienced by DNN models when learning them and label correctness. A novel fine-grained confidence modeling (FGCM) framework is proposed to cluster samples into these five categories; with each cluster, FGCM decides whether to accept the cluster data as they are, accept them with label correction, or accept them as unlabeled data. By applying different strategies to the fine-grained clusters, FGCM can better exploit training data than previous methods. Extensive experiments on widely used benchmarks CIFAR-10, CIFAR-100, clothing1M, and WebVision with different ratios and types of label noise demonstrate the superiority of our FGCM.

Deep-learning models require large amounts of accurately labeled data. However, for medical image segmentation, high-quality labels rely on expert experience, and less-experienced operators provide noisy labels. How one might mitigate the negative effects caused by noisy labels for 3D medical image segmentation has not been fully investigated. In this paper, our purpose is to propose a novel hybrid robust-learning architecture to combat noisy labels for 3D medical image segmentation. Our method consists of three components. First, we focus on the noisy annotations of slices and propose a slice-level label-quality awareness method, which automatically generates label-quality scores for slices in a set. Second, we propose a shape-awareness regularization loss based on distance transform maps to introduce prior shape information and provide extra performance gains. Third, based on a re-weighting strategy, we propose an end-to-end hybrid robust-learning architecture to weaken the negative effects caused by noisy labels. Extensive experiments are performed on two representative datasets (i.e., liver segmentation and multi-organ segmentation). Our hybrid noise-robust architecture has shown competitive performance, compared to other methods. Ablation studies also demonstrate the effectiveness of slice-level label-quality awareness and a shape-awareness regularization loss for combating noisy labels.

Although artificial neural network models have produced very successful results in the time series forecasting problem in recent years, an outlier or outliers in the data set adversely affect the forecasting performance of the artificial neural network models. Dendritic neuron model artificial neural networks which are the most similar neural network model to an artificial neural network model are also adversely affected by outliers in the data set like many artificial neural network models in the literature. In this study, to prevent the dendritic neuron model artificial neural networks from being affected by the outliers in the data set; a robust learning algorithm based on Talwar's m estimator, median statistics to prevent the effect of outliers in the inputs, and a new data pre-processing method are used together in a network structure. In addition, the training of the proposed artificial neural network model is carried out with the symbiotic organism search algorithm. To evaluate the performance of the proposed method, analyses are carried out over the closing prices of the time series of Spain, Italy and German stock exchanges in certain years. According to the results of the analysis of the time series of the relevant stock exchanges, both in their original state and by injecting outliers into the time series, the proposed method has superior forecasting performance even when the time series contains outliers and does not contain outliers. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) becomes challenging in real-world scenarios, which requires learning informative and discriminative features from challenging datasets to obtain robust facial expression recognition. In this paper, we propose an Informative and Discriminative Semantic Features Learning (IDSFL) network for FER against occlusion and head pose in the wild. Specifically, IDSFL aims to mine informative and discriminative semantic features from both low and high levels learned features to learn robust representations. First, a multi-channel feature (MCF) modulator incorporating low-level Gabor features is introduced to learn informative semantic features by capturing adequate diverse and detailed information. Additionally, a specific emotion-aware (SEA) module is proposed to learn discriminative semantic features by aggregating high-level emotion-specific features to focus on each expression category. Thus, IDSFL can collaboratively learn informative and discriminative representations. Extensive experiments on challenging in-the-wild datasets, including RAF-DB, FERPlus and AffectNet-7, demonstrate that our proposed method outperforms most state-of-the-art FER methods. • An effective IDSFL network is presented to learn robust representations for FER in the wild. • A novel multi-channel feature modulator incorporating Gabor features is proposed to learn informative features. • A specific emotion-aware module is used to learn discriminative semantic features for specific expression categories. [ABSTRACT FROM AUTHOR]