Search

We address the discrete inverse conductance problem for well-connected spider networks, that is, to recover the conductance function on a well-connected spider network from the Dirichlet-to-Neumann map. It is well-known that this inverse problem is exponentially ill-posed, requiring the implementation of a regularization strategy for numerical solutions. Our focus lies in exploring whether prior knowledge of the conductance being piecewise constant within a partition of the edge set comprising a few subsets enables stable conductance recovery. To achieve this, we propose formulating the problem as a polynomial optimization problem, incorporating a regularization term that accounts for the piecewise constant hypothesis. We show several experimental examples in which the stable conductance recovery under the aforementioned hypothesis is feasible. [ABSTRACT FROM AUTHOR]

Clustering, as a fundamental exploratory data technique, not only is used to discover patterns and structures in complex datasets but also is utilized to group variables in high-dimensional data analysis. Dimension reduction through clustering helps identify important variables and reduce data dimensions without losing significant information. High-dimensional image datasets, such as Persian handwritten images, have numerous pixels, making statistical inference difficult. Such high-dimensionality property pose challenges for analysis and processing, requiring specialized techniques like clustering to extract information. Incorporating response variable information enhances clustering analysis, transforming it into a supervised method. This article evaluates a supervised clustering approach using Ridge and Lasso penalties, comparing them in analyzing a real dataset while identifying important variables. We demonstrate that despite choosing a small number of variables as important variables, Lasso penalty performs relatively well in predicting the labels of new observations for this multi-class dataset. [ABSTRACT FROM AUTHOR]

This article studies the projection test for high-dimensional mean vectors via optimal projection. The idea of projection test is to project high-dimensional data onto a space of low dimension such that traditional methods can be applied. We first propose a new estimation for the optimal projection direction by solving a constrained and regularized quadratic programming. Then two tests are constructed using the estimated optimal projection direction. The first one is based on a data-splitting procedure, which achieves an exact t-test under normality assumption. To mitigate the power loss due to data-splitting, we further propose an online framework, which iteratively updates the estimation of projection direction when new observations arrive. We show that this online-style projection test asymptotically converges to the standard normal distribution. Various simulation studies as well as a real data example show that the proposed online-style projection test retains the Type I error rate well and is more powerful than other existing tests. for this article are available online. [ABSTRACT FROM AUTHOR]

In order to simultaneously maintain the ship magnetic field modeling accuracy, reduce the number of coefficient matrix conditions and the model computational complexity, an improved composite model is designed by introducing the magnetic dipole array model with a single-axis magnetic moment on the basis of the hybrid ellipsoid and magnetic dipole array model. First, the improved composite model of the ship's magnetic field is established based on the magnetic dipole array model with 3-axis magnetic moment, the magnetic dipole array model with only x-axis magnetic moment, and the ellipsoid model. Secondly, the set of equations for calculating the magnetic moments of the composite model is established, and for the problem of solving the pathological set of equations, the least-squares estimation, stepwise regression method, Tikhonov, and truncated singular value decomposition regularization methods are introduced in terms of the magnetic field, and generalized cross-validation is used to solve the optimal regularization parameters. Finally, a ship model test is designed to compare and analyze the effectiveness of the composite and hybrid models in four aspects: the number of coefficient matrix conditions of the model equation set, the relative error of magnetic field fitting, the relative error of magnetic field extrapolation, and the computational time complexity. The modeling results based on the ship model test data show that the composite model can be used for modeling the magnetic field of ships, and compared with the hybrid model, it reduces the number of coefficient matrix conditions and improves the computational efficiency on the basis of retaining a higher modeling accuracy, and it can be effectively applied in related scientific research and engineering. [ABSTRACT FROM AUTHOR]

For a general domain Ω ⊂ R 3 , we examine the initial value problem (IVP) and final value problem (FVP) for a class of the generalized Kuramoto–Sivashinsky equations (GKSEs). By considering the IVP of GKSE, we prove the global well-posedness theory (local existence, continuation and global existence or finite-time blow-up) in W 1 , p (Ω) for p ≥ 2 . For studying the FVP of GKSE, the problem is severely ill-posed. We suggest the Fourier truncation method to regularize the problem. The well-posedness properties (existence and regularity) in the space W 1 , p (Ω) of the regularized solution are proved. Moreover, the error estimate in W 1 , p (Ω) , p ≥ 2 is represented to verify the convergence of the regularization method when the sought solution belongs to abstract Gevrey spaces. [ABSTRACT FROM AUTHOR]

We consider sparse Markov regime-switching vector autoregressive (MSVAR) models in which the regimes are governed by a latent homogeneous Markov chain. In practice, even for moderate values of the number of Markovian regimes and data dimension, the associated MSVAR model has a large parameter dimension compared to a typical sample size. We provide a unified penalized conditional likelihood approach for estimating sparse MSVAR models. We show that our proposed estimators are consistent and recover the sparse structure of the model. We also show that, when the number of regimes is correctly or over-specified, our method provides consistent estimation of the predictive density. We develop an efficient implementation of the method based on a modified Expectation-Maximization (EM) algorithm. We discuss strategies for estimation of the number of regimes. We evaluate finite-sample performance of the method via simulations, and further demonstrate its utility by analyzing a real dataset. [ABSTRACT FROM AUTHOR]

This work presents a comparative analysis of various machine learning (ML) methods for predicting item difficulty in English reading comprehension tests using text features extracted from item wordings. A wide range of ML algorithms are employed within both the supervised regression and the classification tasks, including regularization methods, support vector machines, trees, random forests, back-propagation neural networks, and Naïve Bayes; moreover, the ML algorithms are compared to the performance of domain experts. Using f-fold cross-validation and considering the root mean square error (RMSE) as the performance metric, elastic net outperformed other approaches in a continuous item difficulty prediction. Within classifiers, random forests returned the highest extended predictive accuracy. We demonstrate that the ML algorithms implementing item text features can compete with predictions made by domain experts, and we suggest that they should be used to inform and improve these predictions, especially when item pre-testing is limited or unavailable. Future research is needed to study the performance of the ML algorithms using item text features on different item types and respondent populations. [ABSTRACT FROM AUTHOR]

Regularization methods in linear regression models with manifest variables have been shown to be effective in selecting key predictors from a set of many variables, while improving predictions for novel observations. Regularization methods are particularly attractive for the analysis of complex multidimensional data when theory development is the primary goal; for example when researchers attempt to predict general or specific factors in bifactor models using many potentially relevant predictors. However, applications of regularization methods in such models are still scarce. In a simulation study, we examined the performance of different regularization methods in bifactor-(S-1) models, varying the number of predictors, the correlations with the outcome (effect size), the underlying structure of multicollinearity as well as the sample size, the type of penalty, and a single-step versus a two-step approach. We explore potential caveats in the use of regularization methods in bifactor-(S-1) models, provide practical recommendations, and discuss future directions. [ABSTRACT FROM AUTHOR]

In this study, a numerical approach is presented to solve the linear and nonlinear hyperbolic Volterra integrodifferential equations (HVIDEs). The regularization of a Legendre-collocation spectral method is applied for solving HVIDE of the second kind, with the time and space variables on the basis of Legendre-Gauss-Lobatto and Legendre-Gauss (LG) interpolation points, respectively. Concerning bounded domains, the provided HVIDE relation is transformed into three corresponding relations. Hence, a Legendre-collocation spectral approach is applied for solving this equation, and finally, ill-posed linear and nonlinear systems of algebraic equations are obtained; therefore different regularization methods are used to solve them. For an unbounded domain, a suitable mapping to convert the problem on a bounded domain is used and then apply the same proposed method for the bounded domain. For the two cases, the numerical results confirm the exponential convergence rate. The findings of this study are unprecedented for the regularization of the spectral method for the hyperbolic integrodifferential equation. The result in this work seems to be the first successful for the regularization of spectral method for the hyperbolic integrodifferential equation.

Regularization regression techniques are widely used to overcome a model's parameter estimation problem in the presence of multicollinearity. Several biased techniques are available in the literature, including ridge, Least Angle Shrinkage Selection Operator (LASSO), and elastic net. In this work, we study the performance of the classical LASSO, adaptive LASSO, and ordinary least squares (OLS) methods in high-multicollinearity scenarios and propose some new estimators for estimating the LASSO parameter "k". The performance of the proposed estimators is evaluated using extensive Monte Carlo simulations and real-life examples. Based on the mean square error criterion, the results suggest that the proposed estimators outperformed the existing estimators. [ABSTRACT FROM AUTHOR]

In this paper, we study inverse source for diffusion equation with conformable derivative: CoD(γ) t u − ∆u = Φ(t)F(x) where 0 < γ < 1, (x, t) ∈ Ω × (0, T). We survey the following issues: The error estimate between the sought solution and the regularized solution under a priori parameter choice rule; The error estimate between the sought solution and the regularized solution under a posteriori parameter choice rule; Regularization and Lp estimate by Truncation method.

In this paper, the problem we investigate is that the inverse problem of identifying the initial value for fractional pseudo-parabolic equation. This problem is ill-posed, i.e. the solution (if exists) does not depend on the measurable data. We give the estimate of conditional stability under an a-priori bound assumption for exact solution. Two regularization methods are used to solve this problem, and under an a-priori and an a-posteriori selection rule for the regularization parameter, the error estimates for these methods are obtained. Finally, several numerical examples are given to prove the effectiveness of these regularization methods. [ABSTRACT FROM AUTHOR]

This work presents a comparative analysis of various machine learning (ML) methods for predicting item difficulty in English reading comprehension tests using text features extracted from item wordings. A wide range of ML algorithms are employed within both the supervised regression and the classification tasks, including regularization methods, support vector machines, trees, random forests, back-propagation neural networks, and Naïve Bayes; moreover, the ML algorithms are compared to the performance of domain experts. Using f-fold cross-validation and considering the root mean square error (RMSE) as the performance metric, elastic net outperformed other approaches in a continuous item difficulty prediction. Within classifiers, random forests returned the highest extended predictive accuracy. We demonstrate that the ML algorithms implementing item text features can compete with predictions made by domain experts, and we suggest that they should be used to inform and improve these predictions, especially when item pre-testing is limited or unavailable. Future research is needed to study the performance of the ML algorithms using item text features on different item types and respondent populations.

Objective: This study aims to compare the accuracy, reliability, and validity levels of the techniques by using various performance measures applying logistic regression models based on regularization approaches from data mining classification techniques on a dataset. Material and Methods: With the development of computerization and technology, machine learning is used in many fields as well as in the field of medicine. It has grown in popularity, particularly in cancer diagnosis. A urine biomarkers dataset from the public platform Kaggle database, which is freely available to all researchers, was used to reveal the most appropriate model for diagnosing patients' pancreatic ductal adenocarcinoma (PDAC). Because of the multicollinearity, the following regression models were considered to classify the disease diagnosis: Logistic lasso, logistic ridge, logistic elastic net, logistic adaptive lasso, logistic adaptive elastic net, and logistic adaptive group lasso. The classification success of the methods used was compared using reliability and validity criteria. Results: There were three statistically significant variables in all logistic regularization models, according to PDAC diagnostic results. Compared to the estimated model results, the logistic adaptive group lasso regression model appears to perform better in PDAC diagnosis. In addition to the three variables in this model, the variables age and plasma CA19-19 have been identified as important variables in PDAC diagnosis. Conclusion: As a result of comparative analyses, the logistic adaptive group lasso regression model outperformed the others in terms of performance measures. [ABSTRACT FROM AUTHOR]

Regularization regression techniques are widely used to overcome a model’s parameter estimation problem in the presence of multicollinearity. Several biased techniques are available in the literature, including ridge, Least Angle Shrinkage Selection Operator (LASSO), and elastic net. In this work, we study the performance of the classical LASSO, adaptive LASSO, and ordinary least squares (OLS) methods in high-multicollinearity scenarios and propose some new estimators for estimating the LASSO parameter “k”. The performance of the proposed estimators is evaluated using extensive Monte Carlo simulations and real-life examples. Based on the mean square error criterion, the results suggest that the proposed estimators outperformed the existing estimators.

In this paper, we consider the problem of finding the initial distribution for a biparabolic equation with logarithmic nonlinearity source. The problem is severely ill‐posed in the sense of Hadamard. Under the a priori assumption on the exact solution belonging to a Gevrey space, we consider a generalized nonlinear problem by using the Fourier truncation method to obtain rigorous convergence estimates in the norms on Lq space with some q ≥ 2 and the Sobolev space Ws, ℓ for some constants s > 0, ℓ ≥ 1. [ABSTRACT FROM AUTHOR]

Vector autoregressions (VARs) and their multiple variants are standard models in economic and financial research due to their power for forecasting, data analysis and inference. These properties are a consequence of their capabilities to include multiple variables and lags which, however, turns into an exponential growth of the parameters to be estimated. This means that high-dimensional models with multiple variables and lags are difficult to estimate, leading to omitted variables, information biases and a loss of potential forecasting power. Traditionally, the existing literature has resorted to factor analysis, and specially, to Bayesian methods to overcome this situation. This paper explores the so-called machine learning regularization methods as an alternative to traditional methods of forecasting and impulse response analysis. We find that regularization structures, which allow for high dimensional models, perform better than standard Bayesian methods in nowcasting and forecasting. Moreover, impulse response analysis is robust and consistent with economic theory and evidence, and with the different regularization structures. Specifically, regarding the best regularization structure, an elementwise machine learning structure performs better in nowcasting and in computational efficiency, whilst a componentwise structure performs better in forecasting and cross-validation methods. [ABSTRACT FROM AUTHOR]

In this work we propose and compare multiple approaches for the formulation of boundary optimal control problems constrained by PDEs. In particular, we define a property of balanced regularity that is not satisfied by traditional approaches. In order to instead guarantee this property, we consider the use of other regularization terms, one involving fractional Sobolev norms and the other one based on the introduction of lifting functions. As required by the fractional norm approach, we present a semi-analytical numerical implementation of the fractional Laplacian operator. All the proposed formulations are also considered in conjunction with constraints of inequality type on the control variable. Numerical results are reported to compare all the presented regularization techniques. [ABSTRACT FROM AUTHOR]

A general topological approach is proposed for the construction of converging artificial neural networks (ANN) by applying decision-making algorithms tuned on a sequence of iterations of continuous mappings (ANN layers). The mappings are selected using optimization principles underlying ANN training, and decision-making based on the results of training a multilayer ANN corresponds to finding a sequence converging to a fixed point. It is found that problems of this class are computationally unstable, which is caused by the phenomenon of dynamic chaos associated with the ill-posedness of the problems. Stabilization methods converging to stable fixed points of the mappings are proposed, which is the starting point for a wide variety of mathematical studies concerning the optimization of training sets in ANN construction. [ABSTRACT FROM AUTHOR]

We study Markov regime-switching Gaussian autoregressive models that capture temporal heterogeneity exhibited by time series data. Constructing a Markov regime-switching model requires making several specifications related to the state and observation models. In particular, the complexity of these models must be specified when fitting to a data set. We propose new regularization methods based on a conditional likelihood for simultaneous autoregressive-order and parameter estimation, with the number of regimes fixed. We use a regularized Bayesian information criterion to select the number of regimes. Unlike existing information-theoretic approaches, the proposed methods avoid an exhaustive search of the model space for model selection, and thus are computationally more efficient. We establish the large-sample properties of the proposed methods for estimation, model selection, and forecasting. We also evaluate the finite-sample performance of the methods using simulations, and apply them to analyze two real data sets. [ABSTRACT FROM AUTHOR]

This work is to investigate the image reconstruction of electrical impedance tomography from the electrical measurements made on an object's surface. An ℓ p -norm (0

In image processing, classical methods minimize a suitable functional that balances between computational feasibility (convexity of the functional is ideal) and suitable penalties reflecting the desired image decomposition. The fact that algorithms derived from such minimization problems can be used to construct (deep) learning architectures has spurred the development of algorithms that can be trained for a specifically desired image decomposition, e.g., into cartoon and texture. While many such methods are very successful, theoretical guarantees are only scarcely available. To this end, in this contribution, we formalize a general class of intersection point problems encompassing a wide range of (learned) image decomposition models, and we give an existence result for a large subclass of such problems, i.e., giving the existence of a fixed point of the corresponding algorithm. This class generalizes classical model-based variational problems, such as the TV-l²-model or the more general TV-Hilbert model. To illustrate the potential for learned algorithms, novel (nonlearned) choices within our class show comparable results in denoising and texture removal. [ABSTRACT FROM AUTHOR]

In classical response surface methodology, the optimization step uses a small number of important factors. However, in practice, experimenters sometimes fit a second-order model without previous experimentation. In this case, the true model is uncertain and the full model may overfit. Here, we use an extensive simulation to evaluate several analysis strategies in terms of their optimum locating ability, and use both simulation and published experiments to evaluate their general prediction facility. We consider traditional (reducing via p-values; forward selection), regularization (LASSO; Gauss-LASSO), and Bayesian analysis methods. [ABSTRACT FROM AUTHOR]

In this paper, we explore the challenges associated with biomarker identification for diagnosis purpose in biomedical experiments, and propose a novel approach to handle the above challenging scenario via the generalization of the Dantzig selector. To improve the efficiency of the regularization method, we introduce a transformation from an inherent nonlinear programming due to its nonlinear link function into a linear programming framework under a reasonable assumption on the logistic probability range. We illustrate the use of our method on an experiment with binary response, showing superior performance on biomarker identification studies when compared to their conventional analysis. Our proposed method does not merely serve as a variable/biomarker selection tool, its ranking of variable importance provides valuable reference information for practitioners to reach informed decisions regarding the prioritization of factors for further investigations. [ABSTRACT FROM AUTHOR]

Nonparametric regression frameworks, such as generalized additive models (GAMs) and smoothing spline analysis of variance (SSANOVA) models, extend the generalized linear model (GLM) by allowing for unknown functional relationships between an exponential family response variable and a collection of predictor variables. The unknown functional relationships are typically estimated using penalized likelihood estimation, which adds a roughness penalty to the (negative) log-likelihood function. In this article, I propose a spectral parameterization of a smoothing spline, which allows for an efficient application of Elastic Net regression to smooth and select eigenvectors of a kernel matrix. The classic (ridge regression) solution for a smoothing spline is a special case of the proposed kernel eigenvector smoothing and selection operator. Extensions for tensor product smoothers are developed for both the GAM and SSANOVA frameworks. Using simulated and real data examples, I demonstrate that the proposed approach offers practical and computational gains over typical approaches for fitting GAMs, SSANOVA models, and Elastic Net penalized GLMs. for this article are available online. [ABSTRACT FROM AUTHOR]

An resilience optimal evaluation of financial portfolios implies having plausible hypotheses about the multiple interconnections between the macroeconomic variables and the risk parameters. In this article, we propose a graphical model for the reconstruction of the causal structure that links the multiple macroeconomic variables and the assessed risk parameters, it is this structure that we call stress testing network. In this model, the relationships between the macroeconomic variables and the risk parameter define a "relational graph" among their time‐series, where related time‐series are connected by an edge. Our proposal is based on the temporal causal models, but unlike, we incorporate specific conditions in the structure which correspond to intrinsic characteristics this type of networks. Using the proposed model and given the high‐dimensional nature of the problem, we used regularization methods to efficiently detect causality in the time‐series and reconstruct the underlying causal structure. In addition, we illustrate the use of model in credit risk data of a portfolio. Finally, we discuss its uses and practical benefits in stress testing. [ABSTRACT FROM AUTHOR]

Vector autoregressions (VARs) and their multiple variants are standard models in economic and financial research due to their power for forecasting, data analysis and inference. These properties are a consequence of their capabilities to include multiple variables and lags which, however, turns into an exponential growth of the parameters to be estimated. This means that high-dimensional models with multiple variables and lags are difficult to estimate, leading to omitted variables, information biases and a loss of potential forecasting power. Traditionally, the existing literature has resorted to factor analysis, and specially, to Bayesian methods to overcome this situation. This paper explores the so-called machine learning regularization methods as an alternative to traditional methods of forecasting and impulse response analysis. We find that regularization structures, which allow for high dimensional models, perform better than standard Bayesian methods in nowcasting and forecasting. Moreover, impulse response analysis is robust and consistent with economic theory and evidence, and with the different regularization structures. Specifically, regarding the best regularization structure, an elementwise machine learning structure performs better in nowcasting and in computational efficiency, whilst a componentwise structure performs better in forecasting and cross-validation methods.

In genetic association studies, pleiotropy is a phenomenon where a variant or a genetic region affects multiple traits or diseases. There have been many studies identifying cross-phenotype genetic associations. But,most of statistical approaches for detection of pleiotropy are based on individual tests where a single variant association with multiple traits is tested one at a time. These approaches fail to account for relations among correlated variants. Recently, multivariate regularization methods have been proposed to detect pleiotropy in analysis of high-dimensional genomic data. However, they suffer a problem of tuning parameter selection, which often results in either too many false positives or too small true positives. In this article, we applied selection probability to multivariate regularization methods in order to identify pleiotropic variants associated with multiple phenotypes. Selection probability was applied to individual elastic-net, unified elastic-net and multi-response elastic-net regularization methods. In simulation studies, selection performance of three multivariate regularization methods was evaluated when the total number of phenotypes, the number of phenotypes associated with a variant, and correlations among phenotypes are different. We also applied the regularization methods to a wild bean dataset consisting of 169,028 variants and 17 phenotypes. [ABSTRACT FROM AUTHOR]

We discuss the convergence of regularization methods for mathematical programs with complementarity constraints with approximate sequence of stationary points. It is now well accepted in the literature that, under some tailored constraint qualification, the genuine necessary optimality condition for this problem is the M-stationarity condition. It has been pointed out, (Kanzow and Schwartz in Math Oper Res 40(2):253–275. 2015), that relaxation methods with approximate stationary points fail to ensure convergence to M-stationary points. We define a new strong approximate stationarity concept, and we prove that a sequence of strong approximate stationary points always converges to an M-stationary solution. We also prove under weak assumptions the existence of strong approximate stationary points in the neighborhood of an M-stationary solution. [ABSTRACT FROM AUTHOR]

Dynamic forces are very important boundary conditions in practical engineering applications, such as structural strength analysis, health monitoring and fault diagnosis, and vibration isolation. Moreover, there are many applications in which we have found it very difficult to directly obtain the expected dynamic load which acts on a structure. Some traditional indirect inverse analysis techniques are developed for load identification by measured responses. These inverse problems about load identification mentioned above are complex and inherently ill-posed, while regularization methods can deal with this kind of problem. However, most of regularization methods are only limited to solve the pure mathematical numerical examples without application to practical engineering problems, and they should be improved to exclude jamming of noises in engineering. In order to solve these problems, a new regularization method is presented in this article to investigate the minimum of this minimization problem, and applied to reconstructing multi-source dynamic loads on the frame structure of hydrogenerator by its steady-state responses. Numerical simulations of the inverse analysis show that the proposed method is more effective and accurate than the famous Tikhonov regularization method. The proposed regularization method in this article is powerful in solving the dyanmic load identification problems. [ABSTRACT FROM AUTHOR]

The estimation of flexural stiffness from static loading test data is the basis of many methods assessing the condition of structural elements. These methods are usually developed under the assumption of having sufficiently accurate data available. Hence, their performance deteriorates as the differences between the measured and true values of the response, often denoted as noise, increase. The proposed methodology is specifically designed to mitigate errors derived from noisy static data when estimating flexural stiffness. It relies on the linearization of the equations relating displacements to stiffness through the unit-force theorem, combined with regularization tools such as L-curve and generalized cross-validation. The methodology is tested using theoretical simulations of the static response of a simply supported beam subjected to a 4-point flexural test for several levels of noise, two types of responses (deflections and rotations), and different levels of discretization. Recommendations for selecting the optimal regularization tool and parameter are provided. The use of rotations as inputs for predicting stiffness is shown to outperform deflections. Finally, the methodology is extended to a statically indeterminate beam. [ABSTRACT FROM AUTHOR]

An adaptive regularization algorithm is proposed that uses Taylor models of the objective of order p, p ≥ 2 , of the unconstrained objective function, and that is guaranteed to find a first- and second-order critical point in at most O (max { ϵ 1 − p + 1 p , ϵ 2 − p + 1 p − 1 }) function and derivatives evaluations, where ϵ 1 and ϵ 2 are prescribed first- and second-order optimality tolerances. This is a simple algorithm and associated analysis compared to the much more general approach in Cartis et al. [Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints, arXiv:1811.01220, 2018] that addresses the complexity of criticality higher-than two; here, we use standard optimality conditions and practical subproblem solves to show a same-order sharp complexity bound for second-order criticality. Our approach also extends the method in Birgin et al. [Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models, Math. Prog. A 163(1) (2017), pp. 359–368] to finding second-order critical points, under the same problem smoothness assumptions as were needed for first-order complexity. [ABSTRACT FROM AUTHOR]

The past few years have witnessed the fast development of regularization methods for deep learning models such as fully-connected deep neural networks (DNNs) and convolutional neural networks (CNNs). Part of previous methods mainly consider to drop features from input data and hidden layers, such as Dropout, Cutout and DropBlocks. DropConnect select to drop connections between fully-connected layers. By randomly discard some features or connections, the above mentioned methods relieve the overfitting problem and improve the performance of neural networks. In this paper, we proposed a novel regularization methods, namely DropFilterR, for the learning of CNNs. The basic idea of DropFilterR is to relax the rule of weight-sharing in CNNs by randomly drop elements in convolution filters. Specifically, we drop different elements in convolution filters along with their moving on input feature maps. Moreover, we may apply random drop rate to further increase the randomness of the proposed method. Also, we find a suitable way to accelerate the computation for DropFilterR based on theoretical analysis. Experimental results on several widely-used image databases such as MNIST, CIFAR-10 and Pascal VOC 2012 show that using DropFilterR improves performance on image classification tasks. [ABSTRACT FROM AUTHOR]

A regularization algorithm using inexact function values and inexact derivatives is proposed and its evaluation complexity analyzed. This algorithm is applicable to unconstrained problems and to problems with inexpensive constraints (that is, constraints whose evaluation and enforcement has negligible cost) under the assumption that the derivative of highest degree is β-Hӧlder continuous. It features a very flexible adaptive mechanism for determining the inexactness which is allowed, at each iteration, when computing objective function values and derivatives. The complexity analysis covers arbitrary optimality order and arbitrary degree of available approximate derivatives. It extends results of Cartis, Gould, and Toint [SIAM J. Optim., to appear] on the evaluation complexity to the inexact case: if a qth-order minimizer is sought using approximations to the first p derivatives, it is proved that a suitable approximate minimizer within ⲉ is computed by the proposed algorithm in at most O (ⲉ−p+β/p−q+β) iterations and at most O(|log(ⲉ)|ⲉ−p+β/p−q+β) approximate evaluations. An algorithmic variant, although more rigid in practice, can be proved to find such an approximate minimizer in O(|log(ⲉ)|+ⲉ−p+β/p−q+β) evaluations. While the proposed framework remains so far conceptual for high degrees and orders, it is shown to yield simple and computationally realistic inexact methods when specialized to the unconstrained and bound-constrained first-and second-order cases. The deterministic complexity results are finally extended to the stochastic context, yielding adaptive sample-size rules for subsampling methods typical of machine learning. [ABSTRACT FROM AUTHOR]

In recent years, deep neural networks have shown significant progress in computer vision due to their large generalization capacity; however, the overfitting problem ubiquitously threatens the learning process of these highly nonlinear architectures. Dropout is a recent solution to mitigate overfitting that has witnessed significant success in various classification applications. Recently, many efforts have been made to improve the Standard dropout using an unsupervised merit-based semantic selection of neurons in the latent space. However, these studies do not consider the task-relevant information quality and quantity and the diversity of the latent kernels. To solve the challenge of dropping less informative neurons in deep learning, we propose an efficient end-to-end dropout algorithm that selects the most informative neurons with the highest correlation with the target output considering the sparsity in its selection procedure. First, to promote activation diversity, we devise an approach to select the most diverse set of neurons by making use of determinantal point process (DPP) sampling. Furthermore, to incorporate task specificity into deep latent features, a mutual information (MI)-based merit function is developed. Leveraging the proposed MI with DPP sampling, we introduce the novel DPPMI dropout that adaptively adjusts the retention rate of neurons based on their contribution to the neural network task. Empirical studies on real-world classification benchmarks including, MNIST, SVHN, CIFAR10, CIFAR100, demonstrate the superiority of our proposed method over recent state-of-the-art dropout algorithms in the literature.

When calculating different profitability measures for a life insurance company, one of the most important parameters to know is the probability of a policy being in force at any given time after the start of risk bearing. These probabilities are given by the survival function. In this paper, we examine data from a Hungarian insurance company, in order to build models for the survival functions of two life insurance products. For survival function estimation based on the unique parameters of a new policy, Cox regression is used. However, not all parameters of a new policy are relevant in estimating the survival function. Therefore, application of model selection algorithms is needed. Furthermore, if the exact effects of the policy parameters for the survival function can be determined, the insurance company can direct its sales team to acquire policies with positive technical results. When traditional model selection techniques proposed by the literature (such as best subset, stepwise and regularization methods) are applied on our data, we find that the effect of the selected predictors for survival cannot be determined, as there is a harmful degree of multicollinearity. In order to tackle this problem, we propose adding the hybrid metaheuristic from Láng et al. (2017) to the Cox regression in order to eliminate multicollinearity from the final model. On the test sets, performance of the models from the metaheuristic rivals those of the traditional algorithms with the use of noticeably less predictors. These predictors are not significantly correlated and are significant for survival, as well. It is shown in the paper that with the application of metaheuristics, we could produce a model with good predicting capabilities and interpretable predictor effects. These predictor effects can be used to direct the sales activities of the insurance company. [ABSTRACT FROM AUTHOR]

High levels of correlation among financial assets and extreme losses are typical during crises. In such situations, investing in few assets might be a better choice than holding diversified portfolios. We show that constraining the sparse ℓ q -norm of portfolio weights automatically controls diversification and selects portfolios with a small number of active weights and low risk, in presence of high correlation and volatility. We highlight the diversification relationships between the minimum variance portfolio, risk budgeting strategies and diversification-constrained portfolios. Finally, we show empirically that the ℓ q -strategy can successfully cope with bear markets by shrinking portfolio weights and total amount of shorting. [ABSTRACT FROM AUTHOR]

Adaptive cubic regularization methods have emerged as a credible alternative to linesearch and trust-region for smooth nonconvex optimization, with optimal complexity amongst second-order methods. Here we consider a general/new class of adaptive regularization methods that use first- or higher-order local Taylor models of the objective regularized by a(ny) power of the step size and applied to convexly constrained optimization problems. We investigate the worst-case evaluation complexity/global rate of convergence of these algorithms, when the level of sufficient smoothness of the objective may be unknown or may even be absent. We find that the methods accurately reflect in their complexity the degree of smoothness of the objective and satisfy increasingly better bounds with improving model accuracy. The bounds vary continuously and robustly with respect to the regularization power and accuracy of the model and the degree of smoothness of the objective. [ABSTRACT FROM AUTHOR]

Information theory concepts are leveraged with the goal of better understanding and improving Deep Neural Networks (DNNs). The information plane of neural networks describes the behavior during training of the mutual information at various depths between input/output and hidden-layer variables. Previous analysis revealed that most of the training epochs are spent on compressing the input, in some networks where finiteness of the mutual information can be established. However, the estimation of mutual information is nontrivial for high-dimensional continuous random variables. Therefore, the computation of the mutual information for DNNs and its visualization on the information plane mostly focused on low-complexity fully connected networks. In fact, even the existence of the compression phase in complex DNNs has been questioned and viewed as an open problem. In this paper, we present the convergence of mutual information on the information plane for a high-dimensional VGG-16 Convolutional Neural Network (CNN) by resorting to Mutual Information Neural Estimation (MINE), thus confirming and extending the results obtained with low-dimensional fully connected networks. Furthermore, we demonstrate the benefits of regularizing a network, especially for a large number of training epochs, by adopting mutual information estimates as additional terms in the loss function characteristic of the network. Experimental results show that the regularization stabilizes the test accuracy and significantly reduces its variance.